16 tests in 3 months with, 4 winners
A global leisure and tourism business offering experiences in destinations around the world had, with their previous programme, managed 12 experiments in 12 months.
In our first three months we ran 16 experiments, had four clear winners and we changed how their business thinks about change.
The before
Their previous agency ran roughly one experiment a month.
Experimentation, when done well, should speed a business up but when every single test requires that much effort to launch then it makes each test scarce and rare and it makes a business hyper-selective about each test. They revert to non-testing behaviour where everyone is trying to filter a backlog of ideas, without supporting evidence, to find the one that will definitely work.
In the business that can feel like being selective and strategic but in practice people are rationing a scarce resource by guesswork alone, so experimentation never really performs, and instead only acts as a gate on release velocity.
Meanwhile the pressure to do stuff on a website doesn't go away. Instead it takes a path of less resistance, goes around the programme and changes go live on instinct and vibes. A new offer here, a new banner there, strikethrough pricing. So you just get lots of traditional and best-practice tactics, all untested, always added and never taken away, and the team with a KPI to grow online sales mostly ends up delivering changes that do not or cannot be shown to do that.
ENTER Another Web Is Possible
We came in as experimentation experts, working alongside a digital product partner who were already providing research and design services for this client.
From the outset we diagnosed that many of the tensions and challenges that the business faced were tied to just how difficult it was to get a test out the door. We set about creating a programme that would break the bottleneck, and that would be radically transparent with the business at the same time so as to really create the culture of learning and experimentation that would bring everyone along on the journey.
We built a new process for ideas. Each one was documented, given a real and falsifiable hypothesis. We published them all openly in a portal accessible by their whole business.
Ideas that were hyper-focused on specific areas or pages were broadened to be testable across the entire estate. Each experiment had plans for how it'd be rolled out were it to be successful. There were business-critical experiments waiting for us when we started, still following the previous process where it took a huge amount of momentum to create one. We folded those into the programme and tested them alongside the other experiments.
In the next three months something incredible happened. Not only did we smash clean through last year's total experiment count, but we brought the whole business along for the ride. Full transparency throughout the full cycle.
They now see where an idea comes from, why we think it might change behaviour and then what happened when we tested it. The whole business gets insight at a pace it had never experienced before with tests designed to answer real questions about what people want, where to show it to them, what the journey should look like, and what motivates them to purchase.
Four winners in three months
Analysis of the journey showed our users ending up in fascinating behavioural loops on the site. Users who hadn't purchased were ending up in areas designed for users who had already purchased and were looking to find their way around. We had multiple questions we wanted to answer about those users. What were they looking for in that area that they weren't finding in the sales journey? Were they trying to plan out their tourism experience? Why do they get stuck in the loop and never go back to purchase once they start looking around in that area?
This insight led to four conclusive, winning experiments, each driving purchases and revenue.
We paid back more than they'll pay for our programme for the entire year with the results of our first experiment, and everything else is a bonus.
Our "Win Rate" is worse
Technically.
We cannot stress enough how important it is to know what exactly is being measured. Four winners in sixteen tests is a lower win rate than their previous agency had. It puts us at 25% and they had a win rate of 33.33%. But the client is overjoyed with us anyway and it's pretty clear why. At one test a month, an inconclusive test is a disaster but for us an inconclusive test is just another day.
By lowering how difficult it was to test, and dramatically increasing how many tests are run, we made it hurt a lot less for a test to not win, and now the client is moving the needle multiple times a month, we're learning from the tests that don't go the way we expect them to and turning them into experiences that really perform for both the business and the customer.
One losing test showed that some tabs we had initially considered to be part of a nav structure was actually performing merchandising work. That finding shaped a whole bunch of follow-up tests.
We actually really love tests that teach us something, but we also know this programme has to make money, so we made the two live in beautiful harmony.
“
Just seeing what's possible, like how big the scale of change can be, I think has been really eye-opening. I didn't fully appreciate the scale of the test you could do within Convert itself versus needing dev effort.
The business stopped guessing
It was really important to us that this worked across the business, as well as for the team that hired us. We've got to convince an entire business to learn and grow together. We do that by giving them stuff, not taking stuff away from them. The increase in velocity came with an open call to the business. "It will be easier for you to get stuff on the website than it ever has been before". We shared an experiment request form across the business and now ideas all flow through the programme because it is the fastest route to a change.
Teams across the business are engaged with testing for the first time, and the customer insights coming out of it have surprised them, changing their view of what a user needs and when.
“
You guys are taking the heavy lifting of quite a lot of stuff. Jon has been giving us what we need and there is a lot of buy-in, a lot of eyeballs... people are really on board with this as they haven't been previously. So it's definitely a big win for us.
here's our favourite part
We presented our programme and process to the wider business. Questions flooded in and praise came down from the C-suite. The client's own summary is hard to argue with:
“
It's very good to see what a CRO process that is robust, streamlined, efficient, and really clean and neat and clear in its approach can do ... it's getting a lot of positive feedback from SLT and C-suite members as well.
Is your testing programme performing?
This case study really shows what you're buying when you partner with Another Web is Possible. Our technology, our experience and our deep understanding of both Conversion Rate Optimisation and how businesses really operate means we do it right.