How and when to use A/B or Multivariate Test Design
When running a testing programme you will have to decide for each test whether it should be run as an A/B test or a multivariate test. The decisions you make when designing your experiments will significantly impact important variables such as the depth of the insights, the speed of testing and delivering winning variations and therefore the impact of your testing efforts.
The 1,024 Variation Test
So when should we run that 1,024 variation multivariate test (Google actually did this) and when would an A/B test be more appropriate? First, let’s consider some of the important variables:
Time to test - The number of variations in your experiment will affect the length that your test needs to run.
Traffic required - Your traffic volumes will also determine how quickly you…
