Goodhart’s law: on not going by the numbers

Charles Goodhart was an adviser to the Bank of England in 1975.  The advice he gave then has been summarized as Goodhart’s law, which has been summarized as follows:

All economic models break down when used for policy.

Charles Goodhart

Charles Goodhart

His version

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.’

Another short version

When a measure becomes a target, it ceases to be a good measure.

Defense Secretary Robert McNamara demanded a graph of numbers that would show whether or not the United States was winning in Vietnam.  Sure enough, the military responded with “body count” figures that showed the Viet Cong were all being killed many times over, but the United States lost the war.

I thought of Goodhart’s Law in connection with President Bush’s No Child Left Behind and President Obama’s Race to the Top programs.  Teachers and schools are judged on the basis of test results.  So the incentive is for teachers to improve, not learning, but the numbers by which learning supposedly is measured.  Dishonest teachers cheat.  Honest teachers have to take time away from teaching the material to teaching how to pass the test.

The aim is evidence-based policy.  The result is policy-based evidence.

As Cory Doctorow explained on Boing Boing:

Once you start measuring GDP as a way of gauging social welfare, people will start to figure out ways to make GDP go up without improving social welfare (say, by swapping dirty financial derivatives).  Once Google starts measuring inbound links as a way of evaluating the importance of web-pages, people will figure out how to increase the inbound links to unimportant pages (splogging, blogspam).  And once you measure fat or calorie content as a proxy for the healthfulness of food, manufacturers will figure out how to decrease fat and calories without making the food more healthful (reducing fat by adding sugar, reducing calories by adding poisonous artificial sweeteners).

The prime example of Goodhart’s Law in action is Soviet economic planning.  Factories were evaluated on the basis of measured output, irrespective of the usefulness of what was produced.  Machinery factories were actually judged on the total weight of the machinery they produced.  That is why there is no substitute for free markets and the workings of supply and demand.  But large corporations often operate like mini-Soviet Unions until reality catches up with them.

W. Edwards Deming, who was possibly the world’s greatest exponent of using statistics to improvement business performance, objected to judging either managers or workers based on numerical goals.  Understand and improve the process, and the numbers will improve, he said, but trying to improve the numbers without understanding the process is an exercise in futility.

targetHere are some links on Goodhart’s law.

GOODHART’S LAW

Goodhart’s Law

Cheating and Goodhart’s Law

The Importance of Goodhart’s Law

Articles by Charles A.E. Goodhart

Tags: , , , ,

6 Responses to “Goodhart’s law: on not going by the numbers”

  1. David White Says:

    This sounds like the open question objection well known to philosophers at least since Bishop Butler’s axiom, “a thing is what it is and not another thing.” If you try to measure goodness in terms of happiness as utilitarians still do, you are left with the open question: ok, this makes me happy, but does it make me good? Unfortunately, no one is paid to bring up the open question objection, but people are paid or otherwise rewarded for dreaming up new ways around Deming’s profound understanding of quality control. There are other sweeping examples from education. Since the 1960s quantitative measures have been used to allow students to grade teachers, the SAT and similar scores have been used to determine admissions, and the student’s GPA is used to determine graduate and job opportunities. All very neat and scientific. The only trouble is that the students quickly learned it was in their interest to give the teachers high ratings, and the professors found themselves looking better if they gave good grades, and the rich were able to afford coaches who taught them to get higher SATs without knowing more. There are no cynical bad guys here, just a cynical culture of people who are so eager to please their boss and their customers that they don’t take time to think.

    Like

  2. Goodhart’s law: on not going by the numbers « Reddotsg's Blog Says:

    […] Goodhart’s law: on not going by the numbers. […]

    Like

  3. reddotsg Says:

    Exactly what is happening here:-
    http://business.asiaone.com/Business/News/Story/A1Story20120815-365526.html
    On the ground, the story is very different.
    So stats don’t always tell the full story.

    Like

    • philebersole Says:

      To be clear, I think good statistics, like good maps, are important tools of understanding. I quote statistics all the time on this web log. You can’t understand anything unless you have a way to compare it with something else. But the map is not the territory, and the problem comes when you think you are changing things on the ground when you are only changing them on the map.

      Like

  4. Education and Goodhart’s Law | Jon Boeckenstedt's Admissions Weblog Says:

    […] of the practical application of this outside higher education (ambulance response time) here.  Or, as Phil Ebersole says elegantly, “The aim is evidence-based policy.  The result is policy-based […]

    Like

  5. GDP is rising | Osmosis Says:

    […] A very good take on Goodhart’s Law and GDP can be found on Phil Ebersole’s blog: […]

    Like

Comments are closed.


%d bloggers like this: