A/B Split Testing vs. Multivariate Testing

When it comes to the design of a website, the ultimate goal for any business is to have a website that is successful at persuading visitors to perform specific, desired actions. These actions, often called conversions, can include tangible behaviors such as ecommerce purchases, newsletter signups, or quote requests. In other cases, these actions are less tangible, and therefore require the assistance of web metrics to determine when a goal has been reached, such as a longer average time on site, or a lower bounce rate. The process of testing and continually improving the design of a web page to lead to more of these desired actions is called landing page optimization.

As with anything that has optimization in the name, there is no one, “right” way to do something. Rather than settle with one design, landing page optimization is a continual process that combines statistics and variants of your website’s design to determine which variant of your website is most successful.

The world of landing page optimization is divided into two camps. A/B split testing & multivariate. Each has its own strengths and weaknesses, and luckily for you, we want to discuss them in more detail. To begin, let’s start with A/B Split testing.

What is A/B Split Testing?

A/B Split testing is a landing page optimization methodology where you split traffic amongst two different versions of a web page. Differences between the two web pages can range from something as small as a different headline or color—to as large as a completely differing layout. Some common examples of when A/B split testing is used include:

• Test two offers or promotions to see which is most desired (Free shipping vs. 10% discount)
• Test two different color schemes (Warm colors vs. cool colors)
• Test two different layouts (One column vs. Two Column)

In A/B testing, a visitor to your site has an equal chance of seeing either version of your website. Visits and conversions are recorded for each variant of your webpage and the results are compiled in a report. Once you have enough data, you are able to determine which of the versions performed best.

from Smashing Magazine

There are of course, pros and cons of using A/B testing:

Pros for Using A/B Testing

• Easy to understand and implement
• Able to test more dramatic changes in layout, design, and content
• Integrated with Google Analytics
• Much less dependent on technology
• Requires less traffic to test successfully

Cons for A/B Testing

• Does not offer as much insight into why one variant performs better
• Can take lots of time and resources
• Only works for specific goals

What is Multivariate Testing?

The difference may seem trivial, but in multivariate testing, the goal is to test multiple sections/elements of a web page to determine which combination of elements leads to the best results. So as opposed to A/B testing where you are testing two variations of one whole page, with multivariate you are testing variations of different sections of a web page. Typically, this results in a greater number of variables and a larger degree of statistical complexity.

from Beehive Strategy

For example, let’s suppose we have a web page that consists of a headline, an image, and an offer. We have two variations of each element: 2 headlines, 2 images, and 2 offers. We want to test the impact that each of these variables has on the conversion rate. Out of the 3 elements listed above, there are 8 potential combinations. That means that the traffic to our website will be split among 8 variations of a webpage. Software will be required to track and report on which variation leads to the best results.

Let’s take a look at the pros and cons of this methodology.

Pros for Using Multivariate Testing

• Able to test more variations
• Better understanding of an elements individual effect on conversion rate
• More data = better statistical analysis
• Requires more traffic to reach statistical significance
• Less flexibility in terms of changes to design and layout

Cons for Multivariate Testing

• Usually requires many more variable combinations to be run than A/B testing
• Requires more traffic to reach statistical significance than A/B testing
• The restrictions of the test setup constrain marketing creativity

We hope this has helped clear up some of the confusion with the two landing page optimization methodologies. If you have any questions or need any clarification, come visit us on our social media profiles.