Metrics Aren’t Measurement

In The Princess Bride, Inigo Mont0ya says to his manager Vizzini, who calls every setback “inconceivable”:

You keep using that word — I do not think it means what you think it means.

See the brief clip at the bottom of this post. By the way, if you’ve never seen The Princess Bride, go rent it. Watch it with your kids or other loved ones.

Metrics are like Vizzini’s “inconceivable.” They rarely mean what you think they mean.

I have a long section in my book devoted to the evils of substitute metrics — a metric you use when you can’t measure what you really want to measure. The substitute metric I’ve run into most often, both in my own work as a department head and in consulting, is “customer satisfaction.” What you really want to measure is usually repurchase intent — and studies have shown that there is limited or even no correlation between CustSat and repurchase intent. I won’t go into more detail here, at least for now; I’ve covered it in other posts, and of course I’d really [shameless plug alert!] like you to buy the book to learn more.

Presentation guru Jan Schultink uses two graphics about Bush, Obama, and job losses to powerfully explore one of the fallacies around metrics — that numbers never lie. Consider: which of the two following charts presents the correct picture of job losses during the past two years?

The first answer: it depends on your political leanings.

The second (and correct) answer: They both do… because they show different metrics!

The metrics seem very similar, and indeed they are drawn from the same data… but a slight, subtle twist on the actual measure determines whether it appears that job losses have gotten worse or better in the past year. (The left-hand graph shows cumulative losses; the right-hand graph shows losses per month.)

The lesson is that you need to examine metrics carefully. Having no metrics is bad… but having bad (incorrect or misleading or substitute) metrics can be worse.

====================

Here’s a six-second clip of Inigo Montoya (Mandy Patinkin) telling Vizzini (Wally Shawn) that “I do not think it means what you think it means”: