Saturday 6th July 2024

How we orchestrate A/B testing for free.

We recently started doing some A/B testing (exciting, right?) but don't have the money or time to spend on buying and implementing a fancy A/B testing platform. So, we're orchestrating it ourselves and most importantly we're getting meaningful insight.

I've used proper tools to do A/B testing before. They all require some sort of setup, some sort of account (which means another password to store and another email list to find yourself on) and a lot of them cost money. There's also the time and opportunity cost of learning to use the platform.

Our website is served with flask and I figured we would just orchestrate the A/B testing via that, with a version A and version B of the website and, importantly markers on the form fills when someone submits the form to let us know which version they came from.

This took around 30 minutes to implement and it works great. It's so easy to analyse the statistics too, because all of our form fills come through to another internal system that we can easily analyse. Don't get me wrong, there are limitations to this approach, but we can easily see which version is most effective at getting people to submit the form, which is 💸 to us.