Blog CRO

A/B Test: The 5 most common mistakes

A/B Test: Test, test, test. A mantra that in the world of marketing we find on every corner. We can even say that large companies owe a large part of their success to including this mantra in their culture.

Just reading this phrase from, nothing more and nothing less, Jeff Bezos gives us an idea of ​​how important it is to experiment:

“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…” 

And as you know, it's not just Amazon that performs thousands of experiments a year. These other companies, which you surely know, also follow the same line:

  • Google: 600.000 experiments in 2020
  • Netflix: In its own blog the company establishes that each change that is considered to be made to the product goes through a rigorous A/B testing process
  • According to Mark twitching mountain, on Facebook there may be around 10.000 experiments being carried out at any given time.

Now, another point to consider is the experimentation mentality/culture of these companies and what you have to consider if you want to enter this world.

The experimentation process is not about performing a single test, and if it did not result in the improvement you expected, conclude that this does not work. 

It is about carrying out as many experiments as possible in search of obtaining small victories that allow the sustainable growth of the company.

Now, let's know the 5 most common mistakes when performing A/B Testing

1. Not performing enough tests

As you read at the beginning of the post, one of the goals of companies like Amazon and Google is to carry out as many experiments as possible. Which in the case of these companies means thousands of tests per month.

With this we do not mean that you have, from now on, to perform a minimum of a thousand tests per month. But aim to carry out as much as possible that your budget and capacity allows.

In addition to quantity, another point to evaluate is quality. It is important to focus on experiments that have the greatest chance of having the greatest impact. To do this, the best thing you can do is classify your hypotheses based on two main criteria:

  • Number of people the test would impact.
  • Ease of implementation.

 With this information in hand, you can easily see which test to perform first. 

2. Not having a clear hypothesis

When taking a test, we should always ask ourselves what exactly we are going to do and what result we expect, based on the research we previously did.

That is why, although the starting point of the experimentation process is ideas, it is important that you turn these into hypotheses. This way it will be easier to determine the result of the test and extract key learnings.

Below I leave you a formula to develop a hypothesis:

If I (complete with what is going to be done) I hope it happens (what improvement I hope) measured by (measuring tool to use)

An example could be:

If I I directly send the person to the /cart page, once they have added an item to the cart, I hope it happens an increase in the conversion rate of 0,25% measured by Shopify sales report

3 Duration

To obtain conclusive data, which have a greater probability of being repeated over time, one of the important elements to take into consideration is the duration of the test.

For this we recommend:

  1. Test by sales cycle: 3 to 4 weeks
  2. Carry out the tests for full weeks. If you started the test on Monday, it should end on Sunday. This way you can detect some variation on a certain day of the week
  3. Take into consideration the temporal elements of your business.
  4. Obtain, taking into account the previous points and the following, 95% of statistical significance 

4. Sample volume

Let's imagine that we carry out an A/B test and we obtain the following results:

  • Control: Number of visitors 1.000. Number of conversions 50
  • Variation 1: Number of visitors 1.000. Number of conversions 70

With these numbers we can say that we have a statistical significance greater than 95%. However, can we ensure that if we implement variation 1 we will obtain the same results? It is very likely not.

This is where the sample volume comes into play to give reliability to the experiment. In addition to the duration, we must aim to obtain a number of conversions greater than 300 in each variation.

Hence the importance that we discussed in point 1 of classifying our hypotheses. By doing so you will see, of the different tests that you can perform, which one has a sufficiently significant sample that allows you to obtain greater reliability in your A/B test. 

5. Refinements

Lastly, what process do we have in place once the test is completed. Simply saying that one variation was the winner is not enough.

We must, at the end of an A/B test, consider the following points:

  • Document the experiment. This way other people in the company can access its information.
  • Define what the main learnings were.
  • What changes would you make to test better and faster?
  • With the data that you now know, what other tests will you perform.

Now that you know 5 mistakes that you should avoid when carrying out an A/B test, tell us, are there any others that you think should be added to the list?

Share on: