A/B testing in eCommerce - leave your competitors behind

Work on eCommerce never ends, and the more technologically advanced your store is, the more you see potential for growth. However, regardless of what stage you are at - it is worth implementing such measures that will identify best practices and areas worth improving at this time. A/B tests will give you such knowledge.

What are A/B tests and why you should conduct them

A/B testing is not rocket science. The principle is simple, but it's very easy to mess up the tests by wrong preparation stage and wrong analysis of the results. As you will see the purpose of A/B testing is already defined at the beginning. However, the purpose of the test itself is only one aspect. Check how the long-term testing will affect your eCommerce:

• higher conversion rate,
• increased sales and engagement,
• lower bounce rate,
• more user data,
• higher average basket value.

But let's start at the beginning.

A/B testing usually involves presenting eCommerce customers with two different versions of specific elements - ads, emails, web pages. These "different versions" are not completely different formats - they differ only in one element, such as headline, CTA, or graphics.

For this type of study, option A will be the original version, presenting the current state. Option B, on the other hand, is precisely the variant with the change, the results of which we compare with option A.

As you can already guess - in A/B testing we introduce one variable. In case of several - you test multidimensionally. For example in A/B test you can check if red or green CTA works better on given LP:

• Text 1, color red
• Text 1, color green

In a multivariate test, on the other hand, you can test both the color of the CTA and its content simultaneously, creating four variations:

• Text 1, color red
• Text 2, color red
• Text 1, color green
• Text 2, color green

It is also important to distinguish between on-site and off-site A/B testing. In the first case we can test elements such as LP, payment forms, banners on HP etc. Off-site testing is mostly about paid ads, but also newsletters and social media posts.

Step by step A/B testing

Now that you know what A/B testing is, let's move on to implementing it. Contrary to expectations, the most difficult stages are not the ones that require technical knowledge. In the following paragraphs we will present some tools that will help you automate and speed up the process. 

However, the initial stages are the most difficult - it is the initial analysis and hypothesis setting that determine the success of a test.

1. Analysis

Before you optimistically start planning A/B testing for your eCommerce, think about what the purpose of the test is for you. What do you want to test, check and why. Finally, set a clearly defined indicator by which you will evaluate the success or failure of the test result. Do you want to measure reach, engagement, or perhaps registrations or transactions?

Once you have determined the goal and indicator, the next step is to determine the site elements to test. In other words, which elements of the website are critical to achieve the goals of the study. Write down addresses of pages with high bounce rate, abandonment rate, not engaging audience, having the lowest conversions, but with conversion potential. The next step is to choose the test element. Among the most common we can include:

• CTA,
• link formats,
• headline,
• graphics on LP,
• description,
• product label,
• color, size, price location,
• product variants,
• terms of delivery,
• social proof.

Remember that only one element is tested - the rest is unchangeable.

2. Formulating a hypothesis

The hypothesis should follow from the objective, and the A/B test itself should be the verification of that hypothesis. Ask yourself what changes could improve the site's performance and why? Maybe the original variant A has an unappealing CTA? Maybe the shopping cart page overwhelms with the amount of text? Your hypothesis should be simple and literal. You can create one based on a diagram: If..., then..., because.

Example:

If I add a discount code to the banner ad, the conversion rate will increase because the extra promotion is an important motivation to buy.

3. Design

You have a goal, a hypothesis, you know which elements you want to test. It's time for the creative part and design. An open mind and lots of ideas are important here. A brainstorming session with your team is always a good way to come up with the most interesting proposals. Write them all down so that the strongest prototypes for implementation emerge at the end.

4. Test

The test step is practically conducted at the end. In this step, the most important thing is to calculate the sample size. By this we mean the number of hits on the eCommerce website, which will be representative and allow you to clearly state that the difference in results between version A and B is not a coincidence.

As we wrote earlier, there are a lot of tools with which you can conduct tests. So if you don't have a team of developers who will implement the code on your website - nothing lost. Be sure to check out these solutions:

• Optimizely - a tool used by the biggest eCommerce players,
• Google Optimize - a free tool that integrates seamlessly with Google Analytics,
• AB Tasty - a tool dedicated to A/B testing for large eCommerce businesses,
• VWO - one of the most popular tools for analysis and testing. Recommended for both large and small eCommerce.

5. Analysis

Once the test is complete, we again return to analysis - this time of the test results. The purpose of this step is not only to come up with a winning version of the test. It also helps you formulate further hypotheses and test ideas. The A/B testing process should be implemented regularly, it's a long-term process that will keep you adjusting your eCommerce and making more and more advanced changes on an ongoing basis.

Finally, it's worth answering the question, how long does A/B testing take?

Here, it all depends on how many users are visiting your eCommerce. The more, the faster you can collect data and draw conclusions. Most often, it's worth spending about 14-21 days for this activity to get reliable results. Remember also that customer activity depends on a given season, so testing variants must take place at exactly the same time.

A/B tests in the context of SEO activities

This point is very important for the success of testing. Google offers tools to create A/B tests, but the search engine itself, in case of detected malpractice, can strongly mess up your eCommerce positioning. What to pay attention to?

First of all, on proper communication with Google robots. Googlebot hitting your eCommerce site collects information about the content to determine its quality and nature. When conducting A/B tests, miscommunication may end up in duplicate content on the site. Such a consequence is dangerous, especially when testing pages with the most external links. Then, the key page may lose its position on many phrases, which previously had a positive impact on conversion and sales. That is why it is so important to mark the canonical version in test variants.

Another important issue you need to pay attention to is not blocking access to the B version for Googlebot. The crawler should have access to exactly the same content as the user. If you block it, then your eCommerce site may be penalized by lowering its position in the search engine.

A good practice in the context of SEO efforts is to use temporary 302 redirects when directing traffic to a variable URL. The 302 redirect will only work for the duration of the A/B test, and the original URL will be retained by the bots in their index.

Finally - clean up after testing. With the end of the test, remove traces of your actions as soon as possible: alternative URLs, tags, or test scripts.

A/B testing - don't repeat these mistakes!

Time is money - in this Polish proverb there is a lot of truth, especially for people running eCommerce. Poorly conducted A/B test is a waste - of resources, time and money. So what to avoid during the testing?

• testing several elements at the same time - the methodological confusion will cause that you will not get the answer which variable actually influences the better result of a given variant,
• Poor selection of pages to test - especially those that have no real impact on conversions,
• too little traffic to conduct the test - conducting a test requires a solid foundation, and this is the research sample. If your eCommerce site is not visited by anyone - there is no point in conducting tests,
• lack of regularity - A/B testing should become a permanent part of your schedule. Only long-term actions will allow for reliable results and conclusions,
• Unconsidered hypotheses - Have you been in the eCommerce industry for years? Do you think you know your site and your customers inside out? Your gut feelings are important, but don't create hypotheses based solely on them. Brainstorm and confront your employees with their ideas,
• simultaneous testing and promotion with advertising - any paid interference during the test may change the eCommerce traffic structure and thus distort the test results.