How Holistic Testing Can Bring You Greater Rewards

Do you have a holistic testing strategy?

Testing is the best way to find out what makes your customers tick, but if you market across multiple digital channels, you must come up with a testing protocol for each channel: email, web, social media, retargeting, SMS etc.

Or do you?

And even though you already know that email is the hub of your digital marketing programme because you can use email addresses to identify and communicate with your customers across all your marketing channels, most email marketers are not performing the basics and testing their emails as stated in latest report from GetResponse “Email Marketing & Beyond: Global Industry Benchmarks 2017”.

What if I could show you that…

Testing via email can drive your multichannel testing programme and deliver insights across your entire customer database, thereby not only saving you time and money in driving traffic but also providing increased conversions sooner. After all, your email database is representative of your target market – your prospects, your first-time purchasers, your regular customers, your loyal customers as well as your lapsing and lapsed customers.

This is Holistic Testing – a new testing methodology by Holistic Email Marketing - which moves beyond single-channel and basic A/B testing to deliver a more powerful set of customer insights that inform, improve and direct your entire marketing programme.

Why the Holistic Testing method is different

Most of the testing that email marketers do today is "ad hoc" testing: one-off, irregular testing of factors that uses no scientific methodology or hypothesis and doesn't build on previous testing results. Email marketers perform these tests with the aim of determining which version delivers a better result. However, in many cases because of the lack of scientific approach to performing the test, the results are not valid and disbelief in testing sets in.

An example of this is a test that has been set up to determine which subject line delivers the highest clickthroughs. No hypothesis has been defined.

Control: Our sale is on now! Get 15% off today!
Variant: Our sale is on today! Get 15% off now!

In contrast, Holistic Testing comprises a series of regular tests, each of which uses a hypothesis and scientific methodology to guide the research and improve its validity. Additionally, Holistic Testing aims not just to learn which version "won" but also to discover long term insights about the audience that you can roll out to other channels.

Hypothesis: Your statement of the presumed problem, your proposed solution and what you expect your test will reveal. Your results will either support or refute your hypothesis. An example hypothesis could be “Loss aversion copy is a stronger motivator towards conversions than benefit-led copy because people hate losing out more than they enjoy benefiting.”

Your hypothesis helps you choose the right factors to test. Most testing focuses on a single element, but you can test multiple factors that help to test your hypothesis, such as determining whether loss aversion or benefit statements drive more conversions.

For the above hypothesis, you could test subject line, copy and CTA simultaneously. You're still only testing one thing – your hypothesis – but you're using multiple factors working together to test that hypotheses. This is a major difference from normal testing.

Using the same Control Subject line and the above hypothesis, this is how a Holistic test may look, testing multiple factors based upon the same hypothesis:

Subject Line:
Control: Our sale is on now! Get 15% off today!
Variant: 15% off today only. Don’t miss out!

CTA:
Control: Get 15% off >>
Variant: Don’t miss out, act now >>

Scientific Methodology: How you will conduct your test and measure the results. It beats just plugging random variables into your testing platform and seeing what happens. Having these guidelines makes your results less vulnerable to chance or unknown factors.

Now that you have defined which is the winning motivator – benefits or loss aversion - you can repeat this test using email a couple more times to ensure the results weren’t due to an anomaly. If all the tests performed support the same findings, then you can roll out the insights to other channels such as your landing pages, PPC ads, Social Media etc.

Three additional bonuses of Holistic Testing:

1. Using email to test and gain insights on your target market can bring more budget and resources to the email marketing department.

You'll be better equipped to persuade management and other marketing teams that email is essential to assisting other channels, not just converting customers (where we already know it's awesome).

A carefully constructed plan of different hypotheses will give you reams of insights and help guide planning for future campaigns.

2.Your email database is made up of your target market so the insights you discover via testing in email will also apply to your other channels.

Suppose your email tests show that your first-time buyers respond more enthusiastically to free shipping without a purchase limit, but your loyal customers – those who bought three or more times in 12 months or who belong to your loyalty club – prefer VIP access and exclusive offers.

You can apply this to your website copy and organisation, to search keywords, PPC campaigns and related landing pages, and in ads and banners, you run on third-party sites in remarketing or network campaigns. Pretty soon, you’ll find more budget is being allocated to the email marketing department as it’s turned into the main testing channel.

3. Holistic testing encourages incremental innovation (AKA the Aggregation of Marginal Gains).

A simple email A/B test is easy to set up and manage in an email and will deliver concrete information that you can build on in future emails. Or, use the newly discovered insights as the starting point on your landing page to incrementally improve it further with your regular website testing tool.

One change might yield only a 1% improvement, but a series of small gains over time could add up to a revolution, and could save you ££ in driving traffic to the website and start you off in a better position to reaping the rewards sooner.

Why use email for Holistic Testing?

1. Testing by email is more cost-effective.

Today, even basic email messaging platforms offer basic plug-and-play testing. But, even if you must pay for testing, the money you will earn from better marketing messages will more than outweigh the cost, not just on a single campaign but throughout your marketing programme.The more you know about your customers, the better you can speak to them.

The more you know about your customers, the better you can speak to them.

Instead of planning a campaign on what your instinct tells you, you can go right to your data and tell your team, "Let's use emotional language for our Mothers' Day promotion across all channels, because our testing in email shows it's more effective than humour."

You'll waste less time and money on running tests via slower channels which rely on additional budget to drive traffic as well as recoup more money because your emails are more appealing, relevant and communicating the same message (albeit potentially different wording and creative), meaning no disconnects. That's a win-win all around.

2. Email testing has a shorter turn-around time.

Maybe you learned about testing when you used it on your website's homepage, landing pages or product pages, and you dread the prospect of having to wait a month or longer until you get statistically significant responses.

Email testing is different because you're already testing your target audience. You can get answers almost immediately, and certainly within a day or two after you send a campaign. Email is a push channel – you don't have to wait for your customers to find you through search or drive them to the site or social media platform with paid traffic.

This means that you start to benefit by having increased conversions sooner rather than later.

Holistic testing in action: 3 experiments

Below are three example email testing hypotheses that you can apply in other channels, including your homepage, landing pages, products pages, search, PPC ads, banner ads and social media.

1. Emotional question will generate more sales than a directive statement

Factors tested: Subject line, CTA, title, copy, landing page.

The result can be applied to headlines on the website, blogs, PPC ads, banner ads, social media and retargeting ads.

2. Double loyalty points will generate more sales than 15% loyalty points.

Factors tested: Subject line, CTA, title, copy, landing page.

What a fantastic discovery this would be! Once assured that this was a consistent result across multiple email tests, you could apply this finding to your loyalty programmes.

3. An emotive image of a person smiling and wearing outfit will generate more sales than image displaying outfit laid out.

Factor tested: Image on email and landing page

Again, you can roll out the results of this email test to your homepage and product pages as well as in banner ads, social media, retargeting ads as well as print ads.

These hypotheses will need multiple tests to ensure the results are not anomalies. This is their beauty. None of them are bound to any specific wording because you are testing motivation, which has been defined within the hypothesis.

Conclusion:

1. Holistic testing yields the greatest insights.

Regular and systematic testing will deliver the greatest insights into your audience and help you understand what works best for your email programme. Going for long-term gains instead of short-term results gives you a solid foundation for consistent performance and incremental innovation.

2. Testing doesn't exist in a vacuum.

Rather, it's part of a cycle of continuous learning, as you can see in the chart below. The hypothesis guides the testing, which generates data you can analyse and learn from. Applying the learnings to appropriate channels can lead to new research interests, which launch new hypotheses, and the cycle starts all over.

3. Holistic testing has one major drawback.

It's addictive! Email gives you so many factors to test (way more than just the subject line) that once you get the hang of testing, you'll want to keep doing it. (But that's a good thing, so carry on.)

As the saying goes, "Always be testing!"