Mantry is a food subscription service targeting men and featuring American-made small batch food products. The experience spans an online application to an in-home experience. This case study examines using multi-variate testing to better choose design variants and product offerings.
Head of Product
Ideation, Mockups, Quantitative Research
2 Months
2014
After Black Friday, Mantry's biggest sales driver every year was Father's Day. Qualitative research had shown that young adults in their 20's and 30's struggled to find a good gift for their father, who often already had everything he wanted. Our special edition Father's Day gift was created around that need, but how best to sell it? Could we optimize the checkout funnel and affect cart size through multivariate testing before our prime season (end of May to early June)?
With a limited period of time to test (a couple months), a limited amount of test traffic, and a limited amount of engineering time available, tests needed to be prioritized. We had streamlined a lot of the pain points through usability testing, but qualitative methods could only suggest the direction we might take for enhancements - we needed to prove the upside using quantitative methods.
I decided we could probably try to achieve two things simultaneously in the time left and given the expected amount of traffic:
Starting with the homepage, I wanted to test how we might get more users to click through to the subscription page. This would be measured by the number of users who either clicked the CTA in the navigation or on the hero image. The two variables I chose to manipulate were:
These two changes would both appear above the fold, and therefore would be the most likely to influence user behavior. The H1 title would have also been a good option, but seeing as we would change this to be a specific Father's Day message, we couldn't test it until closer to the actual holiday.
User interviews had revealed a couple hints that could guide image selection for the homepage: first, the wooden crate that was included with the gift was a huge differentiating factor. Users saw this as an indication of quality, and it made them feel like they were giving a valuable gift. Secondly, it wasn't always clear exactly what a Mantry was. We had sections of the website that showed everything that was inside alongside descriptions, but not everyone made it that far down the page. Based on these two findings, we chose 3 images that included the wooden crate, and / or the contents inside to test as hero images.
Measured KPI: % clickthrough from home to gift page.
The CTA is one of the most common elements on any website, and even small gains on clickthrough rate can lead to large increases in downstream conversions.
During usability testing, the CTA was sometimes missed, suggesting it could be be made bolder. 6 variants were tested in a multi-arm bandit test, varying message and visual style of the CTA.
Measured Metric: % clickthrough from home to the gift page.
Mantry came in three gift varieties: a single crate, a 3 crate subscription, and a 6 month subscription. We already offered free shipping on the 3 and 6 crate gifts to try to incentivize a higher cart size. Users interviews indicated that while free shipping was nice, saving $10 on shipping for a $225 gift wasn't enough to entice someone who was looking at the single crate option. Users felt like the higher commitment level, and considerably higher price tag warranted a "sweeter deal". Due to margins, offering a deeper discount on longer subscription would likely not be motivating (has 3% off ever moved someone to choose the more expensive option?) We could, however, offer more for the larger gifts by offering a bonus item. Since we were paying for shipping to the giftee's house anyway, adding an bonus item had very little incremental cost. We settled finally on a "jerky kit": consisting of 3 bonus bags of high quality jerky. In exchange for extra exposure, the producer was willing to sell us the jerky for about what it would cost to offer a 3% discount on the gift, but we needed to test the offer to see if it moved the needle enough on our AOV.
I added an icon and text on the gift page for the 3 crate and 6 crate options, which would be hidden and shown with CSS depending on the bucket the user fell into.
Measured KPI: Average Order Value (AOV)
Tests for the hero image and the CTA were set up as multi-armed bandit tests, which have the benefit of:
For testing the bonus gift on a 3 or 6 month subscription, a simple A/B split test used since we only had two variants (control and test), and I was fairly certain the bonus couldn't make things worse (so giving that variant 50% of traffic is a relatively low risk). One important consideration for this test is how it would affect the customer experience. Users shown the bonus gift treatment would need to be tracked so they would receive what was promised to them, and we also had to train our customer service team to honor any requests for the bonus gift. Although Google Optimize should cookie each user and show the same variant any time they visit the site, a user could visit on different machines, clear their cookies, or be told by a friend about the offer.
Tests were run until they reached 95% statistical significance, or until it was clear that they would never converge in a reasonable time.
Over a period of 2 months, the tests outlined above were executed, with the following results:
Test Name
Winning Variant
Change vs. Control
KPI
Home- CTA Test
Red Button Design "Sign Up Now"
+146%
% clickthrough
Home - Hero Image
Control (Products in Open Crate)
0%
% clickthrough
Gift Page - Bonus Gift
Bonus Gift Offer
+36%
Average Order Value (AOV)
Two of the three tests run over the period before Father's Day yielded a significant improvement in the measured KPI. For the "unsuccessful" test, we still learned that the current hero image by far performed the best, and would continue to be the standard to test against. The home CTA test showed a stark improvement - using a red CTA button with the copy "Sign Up Now" performed 146% better than the control treatment (the text "Get a Mantry" beside the company's badge). Most excitingly, the Bonus Jerky kit we offered on the gift page showed a 36% boost in AOV with no change to conversion rate, more than enough to make up the marginal cost of the jerky. Rolling these changes out, and comparing with the Father's Day results from the prior year, we in fact saw this boost in AOV hold. The split for 1, 3 and 6 month gifts is presented below for 2013 and 2014.
Year
1 Month
3 Month
6 Month
2013
65%
31%
4%
2014
35%
61%
4%