How Booking.com A/B Tests like Nobody’s Business

Every year, Google holds a conference focused on sales conversions at their EMEA headquarters in Dublin. The theme for 2017 was “converting mobile consumers,” and two presentations stood out for us in particular: Stuart Frisby from Booking.com & Vanja Mlaco from Transavia. Both companies play in the travel/tourism space and stress the importance of testing culture within their organizations.

“If it can be a test, test it. If we can’t test it, we probably don’t do it.” — Stuart Frisby

The importance of testing

Conversion rate optimization (CRO) is centered around increasing the percentage of mobile/website visitors that turn into customers. A/B testing is meant to help create the best version of a product by specifically tailoring the product towards the consumers’ needs and wants. What appeals to your end consumer, and how exactly do you optimize your app for the largest group of people?

“Go in a direction where your customers are telling you that you’re adding value to their lives.”

Booking.com has been testing for 8 years, and initially built their own tools due to a lack of options on the market at the time. After dedicating funding and resources towards developing their testing processes, they created a strong in-house solution.

According to research by Evercore Group L.L.C., their “testing drives conversions across the whole platform at 2–3 times the industry average.” They are constantly running experiments, whether it be for performance improvements or new product features.

At DevCycle, we, too, recognize the power of user-led growth and testing when it comes to understanding which features actually move the needle. With features like an advanced A/B Testing and Experimentation suite and our Feature-Opt In program, we understand the importance gathering critical user feedback to guide future product decisions.

The process

Booking.com’s process is dependent on the fact that they have the resources to A/B test in-house. Employees are encouraged to come up with hypotheses based on existing data that need to be proved or disproved. There is emphasis on having a hypothesis as opposed to only an idea, as hypotheses are often “more grounded in the reality of the business.” As Stuart put it, “People with ideas are artists, people with hypotheses are designers.” — Tweet This

After the hypothesis is created, the test is broken down into atomic experiments. Testing small iterations ensures less risk, and gives more reliable results since there are fewer interfering factors. In the testing phase, there are 1000 experiments running at any time. To put it into perspective, there are more versions of the Booking.com website live than there are humans that have ever lived. It’s pretty hard to wrap your head around it!

“A/B testing is predicated on the idea that you have to trust your tools…You either need to trust the tools that you have, get better tools, or build your own tools.”

Many companies often wonder whether it’s better to build a testing platform in-house, or outsource to an external solution that supports feature experimentation, such as DevCycle.

Booking.com's decision to build in-house was based on their business model -- they don’t need to worry about supply chain management or maintaining their own inventory, and thus decided to make the necessary investment towards internal testing. However, building a solution of their own was no small feat. They were determined to take on the substantial project of developing an in-house testing platform, and poured in the necessary time and effort. Booking.com made this decision 8 years ago, and it has definitely paid off.

When considering A/B testing, it’s important to evaluate whether your business has enough resources to build out all the software, or if the smarter choice is to look elsewhere for faster results.

Building a culture of experimentation

Creating an A/B testing mindset is impossible without the support and participation of the entire organization. In order for a culture of experimentation to be successful, employee input must be highly valued, and there needs to be as much transparency as possible when it comes to data access.

Booking.com believes in “guidelines, not rules,” and offers employees complete freedom to decide how and what to test. They throw HIPPOs (highest paid person’s opinion) out the window in favour of increased democracy within company decision making. There are 75 front-end product teams each with their own multi-disciplinary components (eg. one designer per team), which allows them to zone in on specific problems. Employees also change teams every 10–12 months in order to see the company through a more holistic view and develop a flexible mindset.

A company that recognizes the power of A/B testing is well-positioned for the future. This is applicable to all industries outside of travel, whether it be sports entertainment or retail e-commerce. At DevCycle, our most successful clients are willing to experiment and back up hypotheses with concrete data using our platform. Booking.com’s approach to optimization is something we hope to see more companies adopting as mobile interactions grow to define the consumer experience.

As Stuart Frisby said:

“Throw out your roadmap. Trust your people. Trust your tools. Trust good science.”

Check out the presentation below:

‍

Create your free DevCycle account here to grow your team's experimentation and A/B testing processes today.