Quick & easy guide to multivariate testing

Creating marketing content—ads, emails, blogs, UIs—without reference to some comparative performance metric is a bit like following a road map to somewhere you’ve never been without looking out the window to see where you are. The world outside may be more complicated and unpredictable than you expect, and it doesn’t matter what the map says if you can’t reach your destination. Marketing successfully requires real-world interaction through testing and variation to adapt content to the subtleties of your audience’s preferences. Testing content with real audiences can apply different degrees of variation, from simple comparisons of two iterations in split and A/B testing to more complex multivariate testing that runs many variables simultaneously.

In one form or another, companies that test marketing content outperform their peers. Of the top 500 e-commerce companies, at least 75% run variable tests on the new marketing content they deliver. In this guide, you’ll learn what multivariate testing is and how it can help you refine large volumes of content quickly and effectively.  

Key takeaways

• Running comparative performance tests on marketing content helps marketers create more engagement and value.

• Marketers call comparative tests that involve only two variables – a control version and a variable – A/B testing. Tests involving more than two variables running concurrently are multivariate tests.

• Multivariate testing yields contextualized insights into higher-performing combinations of variables in content rather than just into individual variables.

What is multivariate testing?

Multivariate testing is a technique for testing the performance of multiple variables in marketing content. Multivariate testing aims to refine multiple variations on content simultaneously to determine the best possible combination for all variables.

image source

Websites, mobile apps and ads contain many modular elements such as colors, fonts, images, CTAs, text boxes and other structural components. Combining these elements in a single webpage or ad creates a unique user experience or perception. It’s impossible to predict precisely which combinations of aesthetic and substantive components will drive the most engagement and elicit the most positive user experiences in audiences of thousands to millions. Multivariate testing focuses on monitoring performance in large variable sets of individual components to reveal higher-performing combinations through real engagement.

Multivariate testing and A/B testing

Multivariate testing resembles A/B testing in that it is fundamentally comparative. Audiences interact with different versions of the same content, and marketers compare the results to determine which performs better. However, A/B tests involve only a control and test variable, isolating the performance value of the variable. 

image source

Multivariate testing differs as it involves more than two variables. Nevertheless, this difference isn’t trivial. Rather, it yields a different kind of insight altogether. A/B testing helps quantify the performance of individual variables. Multivariate testing identifies better performing combinations holistically, revealing how different aesthetic and substantive variations interact in the overall impression created in the audience.

Benefits of multivariate testing

Compared to other testing modes, multivariate testing offers distinct benefits that make it the optimal method in specific circumstances.

1. Eliminates the need for sequential A/B tests

While precisely regarding the performance of specific variables, A/B tests can be time-consuming and often require multiple iterations on the same campaign or page to meet targeted performance goals. Multivariate testing can handle these iterations concurrently, making minor adjustments to continue driving engagement throughout the testing process.

2. Multivariate testing creates more data

A successful multivariate test on an ad or page will involve performance metrics on multiple configurations, creating layers of contextual data for each variable in play. Multivariate testing for marketing teams working with high-traffic channels will create larger troves of actionable data insights than other testing methods.

3. Yields contextual insights

Simpler testing modes such as A/B and split testing can quickly indicate which of two versions of a unit of marketing content outperforms the other. But without more information and variation, you cannot understand why or whether something else affected the underperforming variable in the whole of the content. With multivariate testing, marketing teams can often see positive correlations between seemingly unrelated variables such as a particular CTA paired with a particular image when neither performed well independently. 

Potential limitations of multivariate testing

Marketers employ different content testing modes for a reason. Each is best suited to specific circumstances and constraints. Multivariate testing is no exception and may not be the best choice in every scenario.

1. Requires more time

In A/B tests, 50% of the traffic that encounters the content goes to each version. Thus, in a relatively short amount of time, each variant will get enough traffic to be statistically significant. In multivariate tests containing perhaps dozens of variations, the amount of total traffic needed to generate meaningful data is directly proportionate to the number of variables. For example, if you introduce 20 variations, your test will require 10 times as much traffic to generate the same data quality.

2. May indicate more false positives

The increase in variables and variable relationships in multivariate testing introduces a higher occurrence of what statisticians call familywise error rate (FWER). Familywise errors are false conclusions drawn from hypotheses that passed through untested. In other words, a variable that would underperform in isolation has an increased chance of appearing to deliver better results in the coincidental presence of other variables. Testing to deeper extents in variation will drive down FWER but with added time and expense.

3. Multivariate testing is costlier than other methods

The time and effort necessary to design multivariate tests, the extended duration they require to capture sufficient data and the increased interpretive burden in error elimination all raise the overall cost compared with simpler content testing modes. Multivariate testing may not be the best solution for your needs if time and budgets are major constraints. 

Experiment with A/B & multivariate testing through Optimizely’s web experimentation platform

Take advantage of Optimizely’s fast, comprehensive web experimentation platform to uncover valuable customer insights and create high-performing experiences in short order. With Optimizely, customer sentiment and preference will drive your decision-making, eliminating risky guesswork. 

To get started, contact Optimizely today.