Chapter 9 · Part Three: The Practice
People say you can’t measure brand.
What they mean, almost every time, is that they didn’t decide what to measure before they started.
That isn’t a cheap shot. It’s a description of the standard sequence. The campaign gets approved because somebody senior liked it. It runs. Two weeks before the board meeting, somebody is asked to find out whether it worked, and that person goes looking for numbers that moved during the window. They find some, because numbers always move. And the resulting slide isn’t a measurement, it’s a defense, and everyone in the room can tell, including the person presenting it.
The way out is unglamorous and it’s entirely a matter of order of operations.
What we actually did
Before we spent a dollar at Axonius, we ran a brand recall survey.
Aided recall in our target market: nine percent.
Which is a polite way of saying that ninety-one percent of the people we were trying to sell to didn’t know we existed.
Then we ran the campaign. Then we waited. Then we ran the same survey again. Same instrument, same wording, same market definition.
Six months after the campaign finished: thirty-seven percent.
The six months is the part I care about, and it’s the part that usually gets cut when this story is retold, because it’s the least exciting number in the paragraph.
If you measure while the campaign is running, you’re measuring your own media spend. Of course people have heard of you. You’re on their screen. Measure half a year after you stopped, and you’re measuring whether anybody actually retained anything.
Say the timing before you say the number. “Six months after it finished, thirty-seven” is an argument. “Nine to thirty-seven” is a statistic, and statistics invite the question of what else was going on, which in our case was a great deal, because the company was growing on several fronts at once and no clean causal isolate exists.
There’s a second thing the six-month window does, and I didn’t notice it for years. Aided recall is a memory test. Asking somebody half a year later whether they recognize a name is, literally, measuring whether they remembered. Which means this chapter and the next one are the same chapter viewed from two angles.
Say “aided,” and say it first
Aided recall is a lower bar than unaided. You show somebody a list and ask which names they know, rather than asking them to produce names from nothing.
Name it yourself, out loud, before anybody asks.
I’ve watched marketers get destroyed in a board meeting over exactly this, and never because the number was bad. It goes: awareness went from nine to thirty-seven. Somebody asks which metric. There’s a pause. The pause is the whole event. Now the conversation is about whether you understand your own data, and the number is gone.
Nine to thirty-seven on aided recall is a very large move. It doesn’t need help. Being precise about what it’s costs you nothing and removes the only available attack.
How to run one
The operational version, for somebody who has never commissioned market research and is about to.
1. Decide which question you’re answering. Recall, consideration, and attribute association are three different studies and people conflate them constantly. Recall asks whether they know you exist. Consideration asks whether they would put you on a shortlist. Attribute association asks what they think you’re for. Most companies at the awareness stage of the problem need the first one and buy the third because it sounds more sophisticated.
2. Define the population narrowly, in writing, before you talk to a vendor. This is the part that gets rushed and it determines everything. Security decision-makers isn’t a population. Security decision-makers with budget authority, at companies of one to five thousand employees, in North America, is a population. Write the screening criteria down and keep them identical across waves, because a sample that drifts will produce a lift that’s entirely an artifact of who answered.
3. Write the question once. Then never touch it again, no matter how much better you could word it next time. The comparison is the deliverable, and a comparison across two different questions is worth nothing. Put the exact wording in a document with the date on it.
4. Build the list properly. An aided recall list includes you, your real competitors, a couple of large adjacent names as anchors, and at least one company that doesn’t exist. The fake name is a quality control. A meaningful percentage of respondents in every panel will claim to recognize anything, and the fake name tells you what your over-claim rate is so you can discount accordingly. If your vendor doesn’t suggest this, ask why not, and treat the answer as information about the vendor.
5. Randomize the list order across respondents, or position will do some of your work for you.
6. Baseline before you spend anything. Not before the campaign launches. Before the money moves. The first ad, the first sponsorship, the first pre-announcement to press, all of it contaminates the baseline, and a contaminated baseline is worse than no baseline because it looks legitimate.
7. Re-measure six months after the spend stops. Not during. Not at the end. Six months after.
8. Don’t let the agency that ran the campaign run the study. This shouldn’t need saying. It needs saying.
The cost of this is real but it’s almost never the reason it doesn’t happen. The reason it doesn’t happen is that step six requires you to decide, before you have any evidence, what evidence would embarrass you.
The measurement plan
One page, filled in alongside the Bet Card from Chapter 3.
Objective. The change, in somebody else’s head, stated as a delta.
Baseline. The number today, with the date and the method.
Leading indicators. Things that should move within weeks if this is working. In a memory campaign these are usually behavioral: branded search volume, direct traffic, inbound mentions of the specific idea rather than the company, sales reps reporting the phrase coming back at them.
Lagging indicator. The one that actually settles it, and when you look.
Kill. What number, on what date, means we stop.
Continue. What number means we keep going as is.
Scale. What number means we put more behind it.
Who calls it. One name.
The three-way split at the bottom matters more than it looks. A plan with only a success condition and a failure condition forces a binary at the review, and most real results are neither. Naming the continue case in advance is what stops a merely slow result from being killed by somebody’s impatience, and naming the scale case is what stops a genuine success from being congratulated and then defunded.
Five memory tests that cost nothing
If you can’t fund a study, you can still measure memory. These aren’t substitutes for a recall study at a company that can afford one. They’re better than the thing most companies do instead, which is nothing followed by an argument.
- The repeat-back test. Tell five people outside your company what you do. Wait a week. Ask them what you do. Whatever comes back is your message, whether or not it’s the one you wrote. Run it monthly with different people and track how much of the answer is yours.
- The discovery call count. Listen to twenty recorded first calls. Count how many prospects describe you, unprompted, using language from your messaging. Not the reps. The prospects. This is the cheapest and most brutal number in marketing and almost nobody looks at it.
- The open-text field. Put “how did you hear about us” as a free-text box rather than a dropdown, and actually read the answers every month. A dropdown tells you what you already believed. The box tells you what happened.
- The hallway test. At the next conference, ask ten people in your market if they’ve heard of your company, before you say anything else. Ten isn’t a sample. It’s still more contact with the truth than most marketing leaders have in a quarter.
- The anonymized swap. Your homepage hero, logo removed, shown to ten people in your market. Ask whose it is. You built this in Chapter 2. Run it again after every significant messaging change, and treat “I don’t know” as the finding it is.
The actual lesson
I want to be careful, because the takeaway here isn’t “hire Simone Biles.” Almost nobody reading this can, and it wouldn’t help most of the people who could.
Notice what we measured.
We didn’t measure whether our brand got better. Better is an argument. You can’t settle it, the people who disagree with you’ll never be persuaded, and everybody in the room knows the person presenting has an interest in the answer.
We measured whether people knew who we were. That’s a number. It has a before and an after and a method attached, and the person who disagrees with it has to attack the method rather than your taste.
That’s the belief, and it’s the one I’d defend hardest of the seven:
Better is subjective. Different is measurable.
And if you don’t have a Simone Biles budget, you don’t need one. You need two things that cost nothing: a rule that makes the obvious answer illegal, and a number you write down before you start.
It also quietly resolves the argument this whole book is having. When you make something different, you aren’t asking anybody to agree that it’s superior. You’re making a claim about whether it can be told apart from the alternatives, and that claim can be checked.
Which is the honest reason safe work is so hard to dislodge. Safe work is never wrong, because nobody ever proposed a test it could fail.
The Comfort Check
Chapter 9: proving it worked
1. The comfortable assumption. Brand can’t be measured, so we’ll argue about it afterward.
2. The true objective. Know, on a date we set in advance, whether the market’s belief about us changed.
3. The redesigned approach. One instrument, worded once, baselined before a dollar moves, re-run six months after the spend stops, with kill, continue, and scale numbers written down and one person who calls it.