Table of Contents
In the world of conversion rate optimization, there are primarily three types of tests; sequential, A/B/N, and multivariate. In this guide, you’ll get the rundown on each type of test, which types are best in certain situations, and things to watch out for.
Overview
On the surface, it’s easy to understand the differences between each test type, but you’ll want to be aware of some of the finer details.
Sequential
Sequential tests are run in a sequence, each version of the site is run separately one after another. Performance is measured by comparing past performance of other versions to current performance of new versions.
A/B
A/B tests, also known as A/B, run all the different versions of the page at the same time, most commonly by using testing software. Each visitor coming to the test page may see a different version. The A, B, and N in the name refer to a version A, version B, and an N number of other versions.
Multivariate
Multivariate tests are a more advanced form of A/B/N testing. With A/B/N you typically have a few different versions of a page and each version can change a few things. Multivariate testing allows you to test multiple versions of multiple elements on a page at the same time.
Quick Comparison
| Type | # of Variations | Traffic Level |
|---|---|---|
| Sequential | 1-3 | Low |
| A/B | 2-10 | Medium |
| Multivariate | 10+ | High |
More in-depth
Sequential
Sequential refers to testing different versions of a page one after another during different periods of time. When using this type of testing, one version will run for a given time period, then another will run after it.
When to use it:
This type of test works best when using testing software that isn’t viable and/or when traffic to the test page(s) is low.
Things to consider:
Because you aren’t comparing performance directly and during the same period of time, you need to be particularly aware of the effect any promotions, seasonal changes, or competitors’ actions may have on your traffic over time. Technically, the most you can conclude from a sequential test is “version X was better during time period A, than version Y in time period B.”
An example of an outside factor that can skew a sequential test is if a promotion was run during the time period of one version of the test and not in another. The results may skew toward the first version because of the promotion, not necessarily because the first version is a better performing page.
A/B
The A/B testing method involves testing different versions of a page at the same time and each successive visitor may see a different version.
When to use it:
This test type is the default test run by most testing platforms and for good reason. It’s the easiest way to get statistically valid comparison data without the need for large amounts of traffic.
Things to consider:
Pages with particularly low traffic and/or low conversion rates may take a very long time to reach a statistically valid conclusion. In those cases, it may be necessary to lower the acceptable level of significance or to draw conclusions based on trend analysis.
Multivariate
This method involves testing a number of variations of a few different elements on a page interchangeably. Each additional element or variation increases the number of page versions considerably when using this method.
When to use it:
Use the multivariate testing method when your goal is to refine the design of a page, the test page has a large amount of traffic and there is a relatively high conversion rate.
Things to consider:
Without a substantial amount of traffic and a high conversion rate, a large number of variations can take a very long time to come to a statistically significant result.
Next steps
Now that you know the different test types, you can explore more about validating test data and then putting those results into action to help you reach your goals, whether that be sales, downloads, signups, etc.