Lying With Metrics: A Sad Real-World Example

Today’s NY Times has a story on how El Paso schools lived up to the project management maxim “you get what you measure.”

(Note: It’s the NY Times and thus has a potential paywall.)

There have been too many stories of now school districts, feeling pressured to meet No Child Left Behind goals, manipulated the system. This story talks of something worse, of lives destroyed: students who didn’t measure up were kicked out of the very system that was intended to help them.

The issue isn’t the worth of No Child Left Behind, but the metrics that were left behind. You get what you measure.

I have no doubt the bipartisan intent of the law was to make our kids smarter, especially those who were struggling. Society gets smarter when everyone gets smarter. But look what the law measures. It’s not hard, unfortunately, to see these stories coming.

I’m not sure I know what the right metrics are. (The Freakonomics guys have some suggestions, but there remain lots of issues.) I’m not sure we can measure what we want to measure, which is a relatively amorphous idea that schools that haven’t been very good at educating weaker students are getting better at it. Therein lies the corollary to the first law of metrics: “Substitute metrics drive off-target behavior.”

The Times article is a particularly sad example, but legal metrics demonstrate these laws all the time, whether it’s measuring hours (rather than value or productivity) or measuring potential loopholes. Both are valuable and aren’t totally out of alignment with what clients want… but they don’t directly lead to it, and thus at times work against client interests. (They sometimes work against firm/practice interests as well.)

So remember the first law of metrics.

The First Law of Metrics:

You get what you measure.

The Substitute-Metrics Corollary:

Substitute metrics drive off-target behavior.