Can a new subject line tool eliminate A/B testing?

SmartInsights tool review: Touchstone

Rumour has it that the next Indiana Jones film will see our favourite archeologist searching for the greatest undiscovered treasure of modern times: the perfect subject line.

In the meantime, we're left looking for guidance on how to improve subject line performance by using the "right" words. Do we say "free" or "no cost"? "Summer" or "Summertime"? "Deal" or "Offer"?

Any tool that helps us consistently pick out the best words or phrasing would, of course, have two huge advantages:

• The benefits go straight to your bottom line: subject lines have a big influence on opens, clicks and other success metrics.
• You can skip a lot of A/B subject line testing: A/B tests are not always easy to do right (and can give misleading results if done wrong).

Unfortunately, the tools or reports offering advice on subject line wording tend to suffer from two critical flaws.

First, they commonly base recommendations on a review of past subject lines and the average results associated with words and phrases, irrespective of the context or sender.

So a word might only appear "good" because it's used often by senders who, for example, get high open rates for other reasons (like B2B publications with highly-targeted audiences).

Second, they cannot account for the uniqueness of your own audience and email programme. What works for "them" doesn't always work for you (and vice versa).

The Touchstone tool (beta version) seeks to address both flaws to make truer predictions of how a proposed subject line will perform in practice and to suggest better alternatives.

Its analysis of words in its subject line database doesn't ask "How well do subject lines using this word do?". Instead, it asks "How well do subject lines using this word do compared to other subject lines used by the same sender?" So, all things being equal, it's more likely to capture the incremental effect of wording on success.

Second , it encourages you to upload your campaign history so it can apply its algorithmic analysis to your list and emails.

It works much like you'd expect. You enter a subject line and a few seconds later, Touchstone predicts how your open rate, click rate and delivery rate would likely deviate from the average when compared to your typical campaign, your sector or the entire database:

The tool also highlights suggested improvements based on an analysis of synonyms:

The theory is great, since it tackles those flaws that make subject line optimisation tools problematic. But do the results match the theory?

According to the tool's initiator, Dela Quist: "Touchstone correctly predicted the winner of our Alchemy Worx newsletter A/B split tests 93.3% of the time.  And the results are just as good for larger users: our biggest sender has seen a 93.1% prediction rate and Touchstone predicted the actual open rate within 1% of its true value two-thirds of the time." He expects these numbers to improve further as the algorithm evolves and the data set grows.

Sign-up to the beta test is free (a paid model is likely in the future), so I'd suggest simply testing the tool for yourself. Upload your campaign history, then check how accurately Touchstone predicts the results of your next campaign or subject line test.

Four potential benefits

If the results convince you, here are the advantages I can see:

• You can improve your chances of more opens and clicks in a matter of seconds and, as explained, save the hassle of A/B tests.
• The tool's word change suggestions support creativity, which is great when your copywriting batteries are empty.
• It helps automate the analysis of your own campaign subject lines by using your results as an independent set of data for predictions.
• It helps take the subjectivity out of manual subject line comparisons. So, for example, it's easier to end in-house arguments about subject line approaches and word choice.

If, however, you only look at predictions based on other people's results, you're missing out on the true value of the tool. Your own data is particularly important for exploring variations on your existing subject line approaches and for niche lists where others' results are unlikely to be relevant.

Uploaded data also contributes to the total database and analysis (and, ultimately, accuracy) in what you might call crowdsourced subject line optimisation. Individual subject lines (and their associated senders or metrics) cannot be viewed by users, however.

Is it perfect?

All good so far, but there are things to watch for:

• You still need to use your brain. It's usually possible to "break" any subject line tool by finding some nonsensical subject line that is predicted to do well. But you wouldn't use it. You still have A/B testing as a fallback, too.
• Touchstone will help you build a better subject line step-by-step, but you still need to come up with the initial words yourself of course. At the moment the suggestion tool is also limited to a certain collection of words though I imagine this will expand as the tool moves through beta.
• There is very little support content available right now: I would have liked a little more clarity as to what some of the displays mean to avoid ambiguity or misinterpretation. But again that is likely a beta issue.

All-in-all I'm sold on the concept, particularly if you're a big, regular sender with enough campaign data to enable meaningful analysis of your previous subject lines. The beauty of a tool like Touchstone, of course, is that you can test it out first to gauge its accuracy and reliability: let us know what you think.