How to challenge a number you think is wrong

The fastest way to challenge a number is to ask what it counts. Almost every metric people fight about is computed correctly and measures something nobody meant to measure.

You have seen the thread. Someone posts a chart, someone else types "I don't think that's accurate", and forty messages later the chart is still on the slide.

That message fails because "this is wrong" is a claim about arithmetic, and the disagreement is hardly ever about arithmetic.

Decide which challenge you are actually making

There are four different things people mean by "that number is wrong", and only one of them is a bug. Work out which one you mean before you type anything.

• The math is wrong. Someone summed the wrong column or the date range slipped.
• The definition is wrong. The query does exactly what it was asked to do, and what it was asked to do counts the wrong people.
• The number is fine and the conclusion is not. It went up, and the reason on the slide is not the reason it went up.
• The number is fine and nothing depends on it. It moves, and no decision anywhere is waiting on it.

The last three are arguments about meaning. Send one of them as "this looks wrong" and the owner will check their query, find it correct, and tell you so. You have spent a round trip and made them defensive on the way.

So name the kind of challenge in your first sentence. "I think the query is right, my problem is with what it counts" saves everyone the detour and lowers the temperature before the detail arrives.

Ask for the definition before you argue with it

Ask what one unit of the number is. That single question does more work than any argument you could write.

What counts as an active user. What counts as a resolved ticket. What counts as a churned account. These sound like beginner questions, which is exactly why nobody asks them, and it is why the answers are so often improvised on the spot.

Half the time the answer settles the whole thing, usually in your favor or theirs within two messages. The other half, the answer is the argument, and now you are having it about something specific instead of about vibes.

Three questions that get you there:

• What exactly counts as one of these?
• What is excluded, and when was that decided?
• Over what window, and does the window move?

Keep them as questions. The moment you write "surely this should exclude internal accounts" you have handed the other person a position to defend instead of a definition to state.

Bring a second number

Bring something the other person can go and look at: a different cut of the same data, or a count from a system with no stake in the answer. One rival number does more than a page of argument.

"It doesn't match what I'm seeing" is the weakest sentence in this whole genre. It is unfalsifiable, so the only available reply is a polite version of no.

When I launched Unvent, the number the store hands you is installs, so installs is what I watched. Installs went up.

The install chart was not wrong. It counted installs, accurately, and installs are a real thing. It simply could not answer the only question I had, which was whether anybody used the thing twice.

So I spent a couple of weeks pleased with a chart that was describing curiosity. The number that meant something was smaller and much less flattering: whether anyone came back the next day. Nobody argued me out of the install chart. A second number did.

Independence matters more than quality here. Two flawed measurements that disagree tell you far more than one clean measurement everyone has stopped looking at.

Attach a decision or do not send it

Say what you would do differently if you are right. Most challenges stop one sentence short of that, and that sentence is what turns a challenge into work someone wants to do.

"If this counts trial accounts, then the retention story is fine and we should leave onboarding alone. If it does not, we are about to spend a quarter fixing something that is not broken." Now the owner is not defending a query. They are helping you avoid an expensive mistake, which is a much nicer role.

Here is the part people push back on. If you cannot name a decision that changes, do not send the message at all. Being right about a metric nobody acts on is an expensive hobby, and it costs the person who built the dashboard more than it costs you.

Let them agree without losing

People defend numbers because the number is attached to them. Someone wrote that query on a Friday with a half-clear brief, and now it is on a slide in front of their boss. Write for that person.

The moves are small:

• Credit the work in one honest clause, then get to the point.
• Assume the definition was reasonable when it was chosen, because it usually was.
• Offer your fix as a question, so agreeing does not read as backing down.

All of that removes reasons someone might argue with you about something other than the number.

Know which ones to drop

Plenty of numbers are wrong and harmless. A slightly overcounted page view in a weekly summary is not worth a thread. Let it be wrong.

Save the challenge for numbers that steer money or headcount. Do it on every chart and people start reading your name on a thread and bracing, which is exactly the reaction you cannot afford on the day a number really is pointing the company somewhere.

The best outcome of one of these messages is that somebody writes the definition down where both of you can point at it later. Most metric arguments I have watched were the same argument happening twice, because the first time nobody wrote down what the word meant.