Metrics that mislead: vanity numbers and Goodhart’s law

Vanity metrics are pleasant to see but lead to no decision. Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.

Numbers convince more than words

Numbers look like facts. They seem more honest than opinions, and so they are believed. The trouble is not that numbers lie but that they answer a question you didn’t ask. “Page views doubled” is true, but it isn’t clear what to do about it. This lesson teaches four habits: check whether a number leads to a decision, remember Goodhart’s law, look at groups and don’t believe single digits.

Vanity and actionable metrics

The American entrepreneur Eric Ries, in The Lean Startup (2011), divided metrics into two kinds. Vanity metrics are pleasant to see, only grow and lead to no decision: total page views, followers of all time, likes. Actionable metrics show the link “action → result”, so that you can tell what worked. To that is added a simple test, still the best:

If this number changes by half, what will I do differently?

If there is no answer, the number decorates the report. Examples:

• Page views per month. Doubled? It isn’t clear: did the right people come or random ones, did they find what they were looking for or not. Vanity.
• Channel followers of all time. They include those long gone. Vanity.
• The share of search visitors who reached the form. If it fell, you’ll go and rewrite the page for people from search. Actionable.
• Leads with a working phone number from the right segment. If there are fewer, you’ll go and look into the form and the sources. Actionable.

What sets actionable metrics apart? Most often they are a share or are counted per something (per visitor, per source, per segment), have a denominator and a period and are tied to a decision. Vanity metrics are usually “bare numbers” with no denominator and no decision.

Goodhart’s law

In 1975 the British economist Charles Goodhart, analyzing monetary policy, formulated an observation that is now applied to any measure. In the short wording of the anthropologist Marilyn Strathern (1997) it goes: when a measure becomes a target, it ceases to be a good measure. Donald Campbell made a close observation.

How this looks on a site. Say you chose “the number of leads” as the main metric (the way we chose a North Star in the lesson on growth, where we warned about this) and began to reward its growth: with a bonus, with pride, with a chart on the wall. What will happen? There will be more leads, because the easiest way to grow the number is to remove fields from the form, add a pop-up or promise a gift for a phone number. And the useful leads will grow no more, they will be fewer: “anything for a phone number” and spam will come. A metric that measured value well while no pressure was put on it has stopped measuring it.

Other examples:

• CTR (the share of clicks) grows from an intriguing headline, while the benefit to the reader falls.
• Time on the site grows if you muddle the navigation.
• The number of reviews grows if you pay for them, and stops meaning anything.
• Speed of reply grows if you answer “received” instead of solving the task.

What to do.

1. Put a quality metric beside it: “leads” plus “leads that reached a conversation”. If the first grows and the second falls, you are fudging the number.
2. Don’t reward one number. A reward starts the fudging.
3. Recheck the link between the number and what you want, once a quarter.
4. Change the metric when it has stopped describing value. Metrics aren’t sacred.

Open the diagram. On the first tab you can “press” on the number of leads and see what happens to the real leads. The formulas are simplified: it is an illustration of the law, not a measurement.

A target becomes a bullseye The average and the mix

Pressure on the “number of leads”: 0%

Leads a month	100
Share of real leads	60%
Real leads	60

No pressure: 100 leads, 60 of them real.

The formulas are simplified: pressure raises the number of leads and lowers the share of real ones. The numbers are invented, but the point is not: a measure that is rewarded gets improved by any means.

Other traps

Mixed groups (Simpson’s paradox). On the second tab of the diagram conversion rose in each group but fell overall. The reason is the mix: in week two more people came on phones, where conversion is lower. An overall average mixes groups, and when the mix changes it can say the opposite of what happens inside each. So important measures are looked at by group: by device, by source, by new and returning visitors.

Small numbers. You saw this in the lesson on the ladder in practice and in the lesson on the funnel and cohorts: three against five look like a change of two thirds, but are really random scatter. Don’t draw conclusions from single digits.

Correlation is not cause. If in months when you wrote more in the channel there were more leads, that doesn’t yet mean the channel is the cause: seasonality, advertising or someone’s post could have done it. A cause is tested with an experiment (the next lesson), not by lining up charts.

Average and median. The average is pulled up by rare large values. For time and for sums the median (the middle) is sounder.

Surplus traffic. Counters pick up robots, internal traffic (you and your colleagues) and spam referrals. They distort everything. A counter can filter them out, but the filters have to be switched on and checked.

The time window. You can’t compare a Monday with a Sunday; compare periods of the same structure (a week with a week, a month with the same month of last year) and write down what happened in them (a promotion, a holiday).

Five questions for any number

When you are shown a figure, ask:

1. What exactly was counted? The definition (remember “bounce”).
2. Out of what number? The denominator.
3. For what period? And is it comparable.
4. Compared with what? With a previous period, with another group.
5. Which decision does this number support? If none, it is vanity.

Check yourself: leads doubled, but the share of leads that reached a conversation fell from 60% to 20%. What do you suspect?

That the number of leads was being “fudged”: it became easier to leave a request (a shorter form, bait), and people came who don’t really need you. This is the Goodhart trap: the number rose, the value did not.

Practice

Take apart the sample dashboard and protect your metric.

1. The table. For six measures mark “vanity”, “actionable” or “misleading” and enter which decision it supports.
2. Three rewritten measures. What it was and what you rewrote it as: a share, or a count per something.
3. A safeguard for your project’s main metric: the main action and a second metric beside it.

The check looks at the marks on the measures where the answer is clear, at the form of the rewritten measures and at the safeguard. Whether the safeguard is sensible for your project is for you to decide: imagine the easiest way to “improve” your main number to no purpose, and make sure the second metric would show it.