Which version works better: testing on readers

Testing story versions means comparing texts that differ in one thing. Five readers show what is unclear; an A/B test makes sense only with a lot of traffic.

You’ve written three versions, so what next

You have collected several stories: an ABT, a case study, a founder’s story, a spine, a kishōtenketsu. Which of them works? It is easy to choose “the one I like”, but you like it, and someone else will read it. You can check this, and without great cost. Three questions need deciding: what we compare, how we measure and how we decide.

What we compare

Rule one: change one thing. If versions A and B differ in story, headline and picture, then when one proves better you won’t understand why. Let only the shape of the story differ, and keep the rest identical.

Rule two: compare what answers your question. “Which version is more convincing to someone seeing us for the first time?” is not the same as “which is prettier”.

What we measure

There are two kinds of measurement, and they complement each other.

• What people do. Whether they read to the end. Whether they press your button. These are numbers, but on a small site there aren’t many.
• What people understand. Asking them to “tell me in your own words what this page is about” shows what came across better than any questionnaire. And the question “what did you not believe?” finds the places where the story breaks.

Ask about understanding and trust, not about liking. “Did you like it?” gets a polite “yes” in answer.

Check yourself: why is it better to ask for a retelling than for a rating?

A retelling shows what a person understood and remembered. A “liked it” is more often politeness than information.

A test on five readers

The researchers at Nielsen Norman Group showed long ago that with a small number of attentive participants (around five per group) you find a large share of the understanding problems, roughly 85 percent in Nielsen’s widely cited model, though that figure has been argued over, and each further person adds less and less. This is a qualitative test, and it answers “what is unclear and what isn’t believed”, not “which version is better, in percent”.

How to run it.

1. Find five people from your audience who aren’t friends that know you.
2. Show each one version (one to five of them, the other to the other five).
3. Ask them to read aloud or think aloud. Don’t prompt them.
4. Ask three questions: “What is this page about?”, “What would you do next?”, “What did you not believe?”.
5. Write the answers down word for word, without paraphrasing.

When you need an A/B test, and what makes it risky

An A/B test shows two versions to different visitors of a site at the same time and compares what they do. It is needed when there are many visitors. If there are a few dozen a week, an A/B test won’t help you: the numbers are random. See for yourself.

20 per version50 per version200 per version1000 per version5000 per version

Run 20 tests

The texts in the diagram are identical, and by design 10% of visitors click on each. With small numbers, chance easily creates a “winner”. As the number of visitors grows, such “wins” disappear. That is why you can’t draw a conclusion about a text from twenty visitors.

Here both texts are identical, and by design 10% of visitors click on each. But with a small number of visitors, one of them regularly “wins” by a few points. The more visitors, the more rarely, and that is a good way to remember it: a small difference on small numbers means nothing.

Typical mistakes, which Evan Miller describes in detail in his article “How Not To Run an A/B Test”:

• Peeking. Stopping the test on the day one version pulls ahead. (Statisticians call it optional stopping, and it inflates false wins.)
• Several changes at once. Then it is unclear what worked.
• Too short a run. Weekly swings get mistaken for an effect.
• A threshold set afterwards. Choosing the criterion for winning after seeing the result.

More on numbers, thresholds and the traps of A/B tests comes in the part on metrics.

Check yourself: why can’t you stop a test when one version pulls ahead?

Random swings on small numbers easily produce a temporary “winner”. You have to decide in advance how much data to collect, and wait.

The test plan

A good plan fits on one page and is drawn up before the test.

1. Versions. What is in A, what is in B.
2. What is the same. The rest of the text, the button, the design.
3. What we measure. Whether they read to the end and whether they take the main action.
4. How many readers. Five at least.
5. Questions. Three, as above.
6. A decision rule. With a number: “I choose the version that was retold more accurately, provided at least 3 out of 5 are ready to take a step after it.” Write it down while you don’t know the result.
7. How we tell the readers. We say it is a test of the text, we don’t collect more than we need, and we let people leave.

A word on the last. Testing on people means handling other people’s data, even if only a little. If you note only answers without names, there is no problem. If you collect details about people, there are rules. In the EU and UK, the GDPR needs a lawful basis (often consent) and that you collect no more than you need, and a test that records people’s sessions or sets analytics cookies falls under it and under the ePrivacy rules (PECR in the UK). In the US there is no single law, but state privacy laws such as California’s apply to many businesses, and recording a call or a video session needs the other person’s consent: in a number of states, including California, everyone on the call must agree. Telling people it is a test, and asking before you record, is both the ethical and the simple way. This is general information, not legal advice.

Practice

1. Put the version you’ve chosen for now on the page: twelve words or more, with no guarantees and no “best” or “No. 1”.
2. Draw up a plan to test two versions by the steps above. The working note with the plan won’t be seen on a finished site: it is not shown in the gallery.
3. Choose a decision rule with a number and write it down.

Then, if you like, run the test on real people: five of them and a notebook. Write the results down and go back to the page: it should change.

How this part ends

You have come the whole way from “why stories work” to “how to check that they work”: structure, hero, forms, page, brand, case study, data, scrolling, ethics and testing. Now your site has a voice. Next it has to be found: that is search optimization, social media and measurement, the subject of the following parts of the course.