Averages hide the thing you need

At one point I was responsible for a book of more than fifteen clients, and over a year we cut late payment penalties across that book by about forty percent. That is a true sentence and I have used it, and it is worth being honest about what it does not tell you.

It does not tell you whether every client improved a little or two clients improved enormously and the rest did not move. It does not tell you whether anybody got worse. Those are completely different situations, they call for completely different work next year, and the forty percent is identical in all of them.

The substitution

Every summary figure is built for a purpose, and the purpose is almost always communication. Somebody has five minutes and a portfolio, or sixty six accounts, or a set of business units, and there is no version of that conversation that involves looking at all of them. So you produce a figure. That is a legitimate and necessary act, and the figure is a genuine answer to the question of what happened overall.

Then the meeting continues, and the same figure is used to decide what to do.

That is the substitution, and I have watched it happen dozens of times without anybody in the room noticing, including me. The number that was constructed for one job gets picked up and used for a different one, in the same breath, with no transition. Nobody argues for it. There is nothing to argue with. It is just the number that is on the table.

The reason it matters is that a summary is defined by what it suppresses. An average exists in order to reduce the influence of unusual cases, which is exactly correct if you want to describe a group. But a decision is almost never about the group. It is about which cases to spend next quarter on, and the cases worth spending on are the unusual ones. The summarising step and the decision step want opposite properties from the same figure.

I have never seen this fail dramatically. It fails as a slow misallocation. You keep working on the middle because the middle is what the number is about, and the two accounts that are genuinely in trouble sit inside the average looking like a slightly worse month.

Where I over-corrected

For a while my response was to refuse the summary. Bring the whole list. All fifteen, all sixty six, every unit, and let the decision be made against what is actually there.

That is worse, and it took me embarrassingly long to admit it.

It is worse practically, because nobody reads sixty six rows in a meeting and what they do instead is form an impression from the first six, which is a sample chosen by sort order. But the real problem is what it does to responsibility. Handing somebody the raw list feels rigorous and it is actually a way of declining to have a view. Deciding which cases matter is the work. It is the part I was being paid for, and passing the full detail upward is a transfer of that work disguised as transparency.

The version I now think is right is not a technique. It is that the summary and the exceptions have to arrive attached to each other, and the exceptions have to be named by whoever made the summary, because they are the only person who knows which cases the figure had to flatten to exist. Forty percent, and here are the two that got worse.

That is more work than producing the number and it is not much more.

What I still cannot do

I cannot reliably tell, before looking, whether a given figure is one where the average means something. Sometimes the population really is homogeneous and the mean is a fair description of every member of it. Sometimes it is two populations wearing one label and the mean describes nobody at all.

There is no way to know which you have without going down into the detail, which is the thing the summary was built to save you from. So the honest position is that the summary is only trustworthy to somebody who has already done the work it was supposed to replace, and everyone downstream of that person is taking it on faith.

Mostly that is fine. It is fine in the way a lot of things are fine, which is that the failures are invisible and slow and never get attributed to the number.