Why we decided to build our AB platform — Akkad Bakkad
There are two times in a man’s life when he should not speculate: when he can’t afford it, and when he can
— Mark Twain
In this post, we’ll talk about our thought process behind setting up an AB testing platform at Treebo. But first, some background…
Treebo was founded in 2015 with a vision of delivering good quality consistent stay experience in the budget hospitality space. Changing the face of Indian hospitality is no mean feat. And the technology at our core is the driving force.
Over the past 2 years we have been working on scaling our own direct demand and building a strong brand presence. We have iteratively focused on the basics of consumer products and for a long time didn’t even have an app.
Building a consumer product from bottom up is a double edged sword — on one hand, you have a lot of products out there to use as reference but on the other hand, everyone has an opinion on how a good consumer product should be, so you need to prioritise hard.
There is a very clear trade-off involved here; you want to build as much as possible, but you also want to build it in the best way you can.
Treebo has always had a true startup-like data driven culture since its inception. For a long time the risk of doing an experiment on a wide scale and 100% traffic wasn’t very high. Hence our product prioritisation never allowed for AB tests until the balance in this equation changed.
During our early days, we set off building out some of the must haves; a high performing functional and super-fast website that loads within a second and provides seamless booking experience. And then came our progressive web app (featured at google IO ) again with focus of sub-second load and simplified user experience and finally we also wrapped the PWA to have a basic app on android. We continued trying out numerous experiments to uplift booking experience and improve our conversion.
One fateful friday evening…
One such idea was to display pre-tax prices on our website.
How this works is very simple, we have one of two choices; we either show guests a lower price upfront to get them deeper into the funnel and hope they continue and book anyway (which is what several online hotel booking companies do by the way) or continue showing the full price upfront, the trade-off here is that we appear more way more expensive upfront and lose the top of the funnel.
We decided to try this out, on a Friday night, we switched to pre-tax prices. Not only did this backfire but no one was able to monitor this in real-time and stop the experiment when all signs pointed that it was hurting our conversions, so we went through one of the worst weekends we’d ever seen.
When we huddled together for a detailed root-cause analysis, we were still a bit surprised that this had tanked so badly, how on earth was it working for all the other companies then?
As we tried to dig deeper, there were 3 issues we faced:
- We exposed this to all our traffic at once
- We had no easy way of switching it off
- We had no way of knowing whether the bad weekend was due to this or some other reason altogether
We decided that we needed a way to solve for each of these problems before we invested more time and energy in building and scaling our direct products. With our traffic scaling and our aspirations to scale it further, we had to think twice before running an experiment and a scientific way to try out an idea was becoming essential.
Time to invest in AB had arrived. And hence a robust A/B testing platform ‘Akkad Bakkad’ was born at Treebo.
So again, Why did we need one?
Enough said above. To summarize, being a data driven organisation, we believe in going by what data says. Often times there are things that are impossible to pre-empt basis your guts. We believe, in today’s world, a robust scientific way to try things out is the fastest way to move the needle.
“In God we trust; all others bring data.”
What were our options?
- Client side off-the shelf AB platform
- Server side AB — Build out your own AB platform — from scratch
- Server side AB — Leverage one of the open source platforms and build on top of the same.
Why did we decide to build our AB Platform on top of an open source platform
Our key considerations
- Page speed performance — With our continued and razor sharp focus on delivering best-in-class page speeds, even a few hundred milliseconds of delay due to an AB platform or an experiment in unacceptable.
- Simple and easy way to measure — One of the key challenges associated with AB is to accurately measure test results. Unless we make this aspect simple, we knew adoption across teams at a high speed would be challenging
- Time-to-market: Like any startup, we need everything as of yesterday. Hence it was important to make the right long term choice and at the same time not compromise on time-to-market.
- AB as a service for our varied needs: At Treebo, consumer booking is one of the many things we work on and almost all our teams were facing with similar problems. It was very important for us to solve for this holistically and fast tracking the scientific experimentation journey in other products across enterprise, data-science, property management and all other areas. This was one of the other important reasons behind building out an internal AB as a service rather than using an off the shelf platform purely for the needs of the consumer products.
Client side AB platforms provide the flexibility to move very fast were we as Product Managers aren’t dependent on our engineering teams to build out every single test out. We can use the in-built WYSIWYG editors to launch a test very quickly. They, however, introduce slowness and some client side AB platforms have flicker issues where users may see the first version and then the second version. This could compromise the quality of results itself apart from slowing down page load. For these reasons, client-side AB was not a consideration.
There were some downsides to server side AB as well. The most significant downside was that Product Managers wont be able to create and run experiments in a DIY fashion. Every experiment would need to be engineered by our development team. Another consideration was that a server side AB would add overhead of maintaining another service and any maintenance is an overhead with a lean team.
Moreover, Server side AB generally adds overheads to page load times, so we had to invent a brand new implementation route to ensure load times are not impacted.
With workarounds and inventive solutions to some of the downsides, the balance clearly tilted in the favour of server side AB for our needs at this point in time.
Building something from scratch that would recreate the fundamental infrastructure layer of Sampling Rate, Segmentation, Bucketing, Experiment Assignment at the same time solve for mutual exclusivity and measurements seemed like reinventing the wheel.
After evaluating several open source platforms, we zeroed on Wasabi as an open source platform on top of which we can build our our AB platform.
Some scenarios where we already (or will in the near future) use Akkad Bakkad
- Data experiments: Does this new sorting algorithm on Search Results Page lead to better conversion or increase in CTR on SRP?
- Visual experiments: Does having Quick Book on SRP lead to increase in conversion?
- Flow experiments: Changes to 2 different pages — Two separate end-to-end flows.
- Infrastructure updates: Few users getting redirected to the new data centre and rest of the users going to the existing infra to gauge stability.
- New product launches: Small bucket of users seeing the new m-site
I hope this post gives you an insight into how technology is playing a vital role at Treebo. Keep checking for more on our learnings from the AB tests.
The rockstars behind this product are Krishna Nellutla, Ankur Jain, Veena Ampabathini, Lakshya Ranganath, Santhosh Kumar, Varun Achar & Aditya Shetty
A big thank you to Bonie for contributing to the post.
Follow us for more such posts from Treebo and staying on top of how product and technology is shaping the hospitality industry.