A/B Test: Test, test, test. A mantra that in the world of marketing we find on every corner. We can even say that large companies owe a large part of their success to including this mantra in their culture.
Just reading this phrase from, nothing more and nothing less, Jeff Bezos gives us an idea of how important it is to experiment:
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…”
And as you know, it's not just Amazon that performs thousands of experiments a year. These other companies, which you surely know, also follow the same line:
Now, another point to consider is the experimentation mentality/culture of these companies and what you have to consider if you want to enter this world.
The experimentation process is not about performing a single test, and if it did not result in the improvement you expected, conclude that this does not work.
It is about carrying out as many experiments as possible in search of obtaining small victories that allow the sustainable growth of the company.
Now, let's know the 5 most common mistakes when performing A/B Testing
As you read at the beginning of the post, one of the goals of companies like Amazon and Google is to carry out as many experiments as possible. Which in the case of these companies means thousands of tests per month.
With this we do not mean that you have, from now on, to perform a minimum of a thousand tests per month. But aim to carry out as much as possible that your budget and capacity allows.
In addition to quantity, another point to evaluate is quality. It is important to focus on experiments that have the greatest chance of having the greatest impact. To do this, the best thing you can do is classify your hypotheses based on two main criteria:
With this information in hand, you can easily see which test to perform first.
When taking a test, we should always ask ourselves what exactly we are going to do and what result we expect, based on the research we previously did.
That is why, although the starting point of the experimentation process is ideas, it is important that you turn these into hypotheses. This way it will be easier to determine the result of the test and extract key learnings.
Below I leave you a formula to develop a hypothesis:
If I (complete with what is going to be done) I hope it happens (what improvement I hope) measured by (measuring tool to use)
An example could be:
If I I directly send the person to the /cart page, once they have added an item to the cart, I hope it happens an increase in the conversion rate of 0,25% measured by Shopify sales report
To obtain conclusive data, which have a greater probability of being repeated over time, one of the important elements to take into consideration is the duration of the test.
For this we recommend:
Let's imagine that we carry out an A/B test and we obtain the following results:
With these numbers we can say that we have a statistical significance greater than 95%. However, can we ensure that if we implement variation 1 we will obtain the same results? It is very likely not.
This is where the sample volume comes into play to give reliability to the experiment. In addition to the duration, we must aim to obtain a number of conversions greater than 300 in each variation.
Hence the importance that we discussed in point 1 of classifying our hypotheses. By doing so you will see, of the different tests that you can perform, which one has a sufficiently significant sample that allows you to obtain greater reliability in your A/B test.
Lastly, what process do we have in place once the test is completed. Simply saying that one variation was the winner is not enough.
We must, at the end of an A/B test, consider the following points:
Now that you know 5 mistakes that you should avoid when carrying out an A/B test, tell us, are there any others that you think should be added to the list?
This year 2024 Black Friday has taken us headlong; not only because the competition is increasing and we have to be aware of campaigns 24/7 but because of the peculiarity of this year. But the hard work has provided us with a series of important lessons that can help us reflect on the […]
Pay attention to this! In the world of marketing, the key is not only to be seen, but to be remembered. Because let’s be honest: you can have the best campaign in the world, but if your brand doesn’t leave a mark, the effort is worthless. Today we tell you how 4 branding elements can not only make […]
In the world of eCommerce, the key to staying competitive is maximizing the profitability of your advertising campaigns. But how do you do this without getting lost in a tangle of metrics and platforms? The answer lies in aligning the sales funnel with the right campaigns and platforms. Below, we explain how to structure your campaigns to […]
