Principles and best practice in Multivariate website testing

Many companies are now turning to Multivariate and AB Testing to meet their goals, typically increased conversion rates or revenue per visit.

Use of multivariate and AB testing is certainly going to increase with the Google Website Optimizer recently coming out of beta (see below). To find out more about best practice and future trends in these testing techniques I recently talked to Mark Simpson, Managing Director of Maxymiser, a UK-based supplier of advanced Multivariate testing tools.

This month there is also a great case study out of how Lovefilm uses structured testing to improve it's results .

Principles and differences between AB and multivariate testing

Q1. Can you outline the principle of multivariate testing and how it differs from AB testing. What types of pages is most commonly used on?

Mark Simpson:

A/B testing is a valid way to increase the performance of the page. However, whilst A/B testing provides a clear indication of consumer response to actual content, the process takes several weeks.

Reverting to the top performing content at the end of the trial is no guarantee of sustained conversion uplift: a range of factors from seasonality to competitive offers and offline influences can and will change consumer response in real time. And this is where multivariate testing comes into its own.

The basis of multivariate testing is to statistically define the best converting version of a web page or series of pages in real-time, taking the guesswork and gut feel away from website design and replacing it with an accurate measure of what works and what doesn"€™t for real visitors to websites, while also enhancing the opportunity for creativity and experimentation.

Multivariate testing allows much more ground to be covered and gives a higher probability of increase as you are looking at the relationship between variants in test areas of a page as well as the winning content in each test area. It is possible to cover months worth of A/B testing in a single multivariate test and achieve better conversion uplift results.

At Maxymiser we have tested on a multitude of pages. There are obvious start points like landing pages and entry pages, however, product pages, and results pages provide an equally valid testing arena. Lately we have seen some of our greatest returns for clients on registration and checkout processes, where we see uplifts directly hitting the bottom line.

Benefits from AB and multivariate testing

Q2. What performance uplift is possible? Can you share any results from clients who have agreed to this?

Mark Simpson:

Over the course of the hundreds of tests we have done we have shown that anything is possible as, even the most basic content changes, such as changes to button colours, have proven to deliver significant conversion uplift.

Across the tests we have implemented for clients we have seen uplifts from a few percentage points up to hundreds of percent increase in conversion metrics, however as a rough guide addressing the check-out process will typically deliver an improvement well in excess of 30%, whilst changes to registration processes have delivered over 50% improvement.

The key to success is designing proper, well thought out and well structured testing schedules.

Click here to see examples of multivariate testing results on the case studies area of the Maxymiser site.

Multivariate testing mistakes

Q3. What are the biggest mistakes you think are made when companies start running these types of site optimisation experiments for the first time?

Mark Simpson:

Prior planning is key in running any tests, picking the right conversion metric(s) to optimise to, and the right areas of content to make a difference to those conversion metrics.

Understanding multivariate testing and tools like automated optimisation helps in getting the maximum out of the technique while limiting business risk.

Advantages compared to Google Website Optimizer

Q4. With Google Website Optimiser freely available, what are the benefits of using paid for tools such as yours? Are they only for the largest companies?

Google Website Optimiser is a great tool, indeed we are authorised consultants for it and have used it for simple tests in the past.

The benefits of using paid for tools can be numerous dependant on the structure of the site and design of experiment.

For example, testing multi-step processes, dynamically generated content, optimising to multiple conversion rates and having the safety net of thorough QA and account management are but a few.

Choosing a supplier for AB and multivariate testing

5. If I"€™m selecting a provider to run these types of optimisation projects for me, how should I select the best tool and company? What should I look for?

• Look for flexibility "€“ in methodology and integration to make sure the tool you pick can quickly and easily be set up to run with your content and systems.
• Get the provider to prove their expertise, look for relevant clients, testimonials and case studies.
• Scalability is key "€“ both in the number of tests and also the technological solutions, when you start getting results from multivariate testing it will not be long before you start looking at segmenting your audience and targeting based on behaviour.