Developing a test and learn programme

How to utilise continuous improvements and ‘big bets’ to achieve operational and strategic excellence

As part of any digital transformation agenda, the ability the test and learn gives businesses the opportunity to experiment, iterate and grow skills and competences throughout the organisation. For digital marketers, this opportunity is becoming increasingly important as technology continues to advance and larger companies face new and emerging threats from more nimble, innovative competitors.

To drive meaningful digital transformation at scale, businesses must therefore be open to the adoption of a test and learn culture, which will enable marketers to optimise digital media activation, create first-class digital experiences and develop learning across the organisation. Much of this will be dependent on each organisation’s stage in the digital transformation journey:

Three big digital trends

The importance of developing a test and learn culture is reflected in the numerous digital trends impacting businesses today. However, I’d like to highlight three that I believe are particularly significant and should influence test and learn planning for digital marketers across all types of businesses:

'Big data’ has grown up

The concept of ‘big data’ has been a familiar theme within the marketing world for at least the last five years, and yet with some predicting that 2016 would be the ‘year of the customer’ there has been an increased emphasis on customer-centric marketing, meaning data must be used intelligently to drive results.

Advanced analytics, better consumer profiles and the right market and customer insights are becoming essential in tying marketing campaigns together to create more integrated experiences.

Mobile continues to dominate

Source: Benedict Evans, 2016

More than half of our waking time is spent on media and much of that time is now consumed on mobile devices. Mobile is an increasingly ubiquitous presence in our lives (49% of Millennials check their phones within 5 minutes of waking up in the morning!) – we’re now in a truly ‘mobile-first’ world.

As with big data, mobile advertising has come of age:

• Mobile accounts for over a half of ecommerce traffic and a third of sales
• More than half of Facebook’s base is mobile-only
• App usage (90% of time) dominates browsers in mobile usage

As Benedict Evans from venture capital firm Andreessen Horowitz demonstrated last year, ‘mobile’ no longer means mobile - consumers regularly consume media on mobile devices more within the home than outside, highlighting the importance of developing a first-class mobile customer experience.

'Content shock’ is real

Source: business2community

‘Content shock’ (coined by Mark Schaefer in 2014), is the result of consumers rejecting brand content due to exponentially increasing volumes intersecting our limited consumption capacity. Basically, people are fed up of poor quality content and are finding ways to filter this out so they can consume what really matters to them.

We can see examples of content shock in Facebook’s decision to dramatically reduce organic reach and the continuing rise of adblocking across all device types:

Source: KPCB, 2016

The average person is exposed to over 400 messages a day. Attention is at a premium and brands must respond accordingly by producing genuinely high quality, relevant content to earn this increasingly precious commodity.

Introducing a test and learn approach

Every test and learn programme will differ based on each business’s appetite to testing and their use of digital marketing in general. However one approach I’m currently adopting takes into account two aspects:

1. Continuous improvements – running trials to optimise day-to-day activations)
2. 'Big bets' – to gain operational lessons and strategic insight to apply across the business

According to Rosabeth Moss Kanter writing in HBR in 2006, successful innovators use an ‘innovation pyramid’, with several big bets at the top that get most of the investment; a portfolio of promising mid-range ideas in test stage; and a broad base of early stage ideas or incremental innovations. This concept allows ideas and influence to flow up or down the pyramid.

For the purpose of this post, I thought I’d bring this approach to life by providing examples of two areas of continuous improvements and one ‘big bet’ to demonstrate:

1. a) how these areas relate to the digital trends highlighted earlier in the post
2. b) how these areas could fit into a digital test and learn programme

The following examples are just a sample of many others that could be used and I’ve tried to keep these fairly broad so they can be applied to different businesses, small and large, across B2C and B2B.

Two areas of continuous improvement

1. Mobile optimisation

Although 2016 cannot be regarded as ‘the year of mobile’ (that came and went at least two years ago!), mobile strategies are maturing and it is essential that brands take a ‘mobile-first’ approach to meet consumer expectations. Mobile cannot be a less-than-web experience - it has to be the experience.

Take advantage of micro-moments

Source: Google, 2014

Mobile has fundamentally changed consumer behaviour and as consumers we think, search and buy differently from five years ago. Mobile gives marketers the opportunity to take advantage of what Google refers to as ‘micro-moments’, those intent-rich moments where decisions are made and preferences shaped.

Test and learn opportunity:

Mobile has fractured the consumer journey into hundreds of real-time, intent-driven micro-moments. Consumers are more loyal to their need rather than a brand so it’s important to test being present at the right moments, e.g.:

• Deliver targeted advertising during key events
• Prompt potential consumers ‘in-store’ with targeted promotions
• Provide help and guidance when it’s needed, e.g. specific searches on YouTube or Google

Reach consumers where they’re spending their time

Source: KPCB, 2016

One of the insights that stood out for me from KPCB’s 2016 internet trends was that advertising currently remains over-indexed to legacy media despite the rapid increase in mobile usage.

Test and learn opportunity:

There is an opportunity to increase ad spend in mobile channels to reach consumers where they appear to be spending most of their time. This will not only improve ad performance but potentially provide a competitive advantage over slower-moving rivals who have failed to respond to this trend.

Establish a presence across multiple platforms

Source: Benedict Evans, 2016

Mobile has led to an ‘unbundling’ of the web. We now consume content across browsers and apps, although the trend is moving more and more towards native apps which could signal the death of the hyperlink.

A mobile-first approach means that mobile must be the ultimate experience, with the web becoming merely an add-on (a complete 180° shift from where this was before).

Test and learn opportunity:

Establish a presence across multiple platforms, including responsive/ adaptive design that works across desktop and mobile, as well as native mobile apps. How is your audience consuming mobile content? What channels and platforms work for them? Where are the optimisation opportunities? Mobile apps don’t work for everyone but if there is an opportunity to test without too much risk it may be worth looking into.

2. Measurement and analytics

Effective analysis and insight should underpin everything we work on as digital marketers. Without a thorough understanding of what is and isn’t performing, you will not have the right actionable insights to make correct decisions.

Establish a measurement framework

A measurement framework/ model is a way to structure your thinking, prioritise goals and organise the KPIs and metrics you’ll use to measure performance.

Avinash Kaushik is a leading thought leader on this subject and as he explains:

“The root cause of failure in most digital marketing campaigns is not the lack of creativity in the banner ad or TV spot or the sexiness of the website. It is not even (often) the people involved. It is quite simply the lack of structured thinking about what the real purpose of the campaign is and a lack of an objective set of measures with which to identify success or failure”.

Test and learn opportunity:

Before jumping into a digital campaign or project, consider creating a measurement framework to structure how all of the following work together:

• Business objectives
• Macro and micro goals
• Key performance indicators (KPIs)
• Targets
• Segments

The example above shows how goals, KPIs and segments are flow from the high-level business objectives that have been set out. The key is to understand how this framework might apply to your business and test a similar approach.

Run regular A/B testing to improve relevance

A/B or multivariate testing should form a key part of your ongoing test and learn programme. By experimenting with different types of content versus a current experience across web pages, social channels and/ or apps, you’ll be able to more accurately determine which variant performs better for a given conversion or goal.

Using the data from A/B testing removes guesswork and subjectivity, giving you the confidence to make more informed, data-driven decisions.

Test and learn opportunity:

A/B testing doesn’t have to be a complex or expensive process. Start small and experiment using particular pieces of content that have generated discussion or debate internally.

If you’re testing something more experimental or risky, take ‘controlled risks’ by showing the new content to only a small proportion of the audience (e.g. 10% of traffic). As you build an understanding and confidence in your hypothesis you can begin to increase the scope of your testing ‘landscape’.

One big bet

Dynamic content and personalisation

‘Big bets’ can be anything that has the potential to enhance or optimise the organisation across multiple levels. Unlike the more operational continuous improvements highlighted above, ‘big bets’ often require more planning and investment, but if successful have the potential to future-proof a business.

For this post I’ve chosen to look at dynamic content and personlisation as an example of a ‘big bet’. This is closely linked to all three of the trends highlighted at the start of the post (data, mobile and ‘content shock’) and as I’ve tried to do throughout the post, this idea can apply to large and small businesses alike.

Optimise content to drive action

The essence of personalisation is about using content that is most relevant to the audience in order to generate higher engagement and conversion. One method of doing this is to use dynamic content, essentially showing the same web page to two people but serving different content within that page based on what we know about them:

Test and learn opportunity:

First and third-party data can be used to create more relevant and compelling experiences, and iterative platform testing can be used with or alongside A/B testing tools to learn about what is/ isn’t working. Take the time to find out what type and quality of data you have and run small tests initially to optimise content for different audience groups.

Create personal video and TV experiences

Source: Google/ DoubleClick, 2016

The way people consume content has fundamentally changed. Whilst ten years ago many of us would watch a show live on one television in a living room, the use of many different devices and streaming services such as Netflix mean that we now watch TV shows across a variety of media and it’s therefore become increasingly difficult to reach audiences with traditional methods of advertising.

However, this challenge has given brands a new opportunity: to develop personalised advertising strategies.

The concept of addressable TV advertising enables brands to reach fragmented, time-shifted audiences and to show different ads to different households who are all watching the same programme.

Test and learn opportunity:

Whilst TV is not a marketing channel open to everyone, and the nascent technology involved in addressable advertising has a number of challenges (including effective measurement and achieving reliability at scale), I wanted to include this example to demonstrate how advertising can be smarter through the use of data.

The emergence of addressable advertising may even give smaller businesses the opportunity to experiment with TV advertising in a way that they couldn’t before. Businesses no longer have to book media to reach millions of people. Instead spots can be bought to target specific groups of people watching certain types of programming at certain times of the day.

Deliver tailored advertising with programmatic

Source: Google, 2016

A core goal for marketers should be about producing content and communications that matter to the audience. Programmatic marketing is something we’ve covered previously, but in a nutshell it’s about enabling brands to be responsive to their audience in real-time, with highly relevant messaging and creativity. The objective is to tailor messages to the right person, at the right moment, in the right context.

Test and learn opportunity:

Depending on device, location and weather, content can be delivered programmatically to different audiences and this is something that can be tested across different campaigns.

Google offers an excellent guide to getting started with programmatic marketing, with a useful checklist of key steps:

1. Organise audience insights
2. Design compelling creative
3. Execute with integrated technology
4. Reach audiences across screens
5. Measure the impact

Summary

With so many major trends beginning to take shape and impact the work we do as marketers, we must look to adapt and find the best ways to take advantage of these changes.

Whilst it would be both unwise and costly to carry out tests for every trend that arises, an effective test and learn programme can enable us to carry out tactical and strategic experiments to build learnings and help us understand what works for our business.