Charles Goodhart was an adviser to the Bank of England in 1975. The advice he gave then has been summarized as Goodhart’s law, which has been summarized as follows:
All economic models break down when used for policy.
Charles Goodhart
His version
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.’
Another short version
When a measure becomes a target, it ceases to be a good measure.
Defense Secretary Robert McNamara demanded a graph of numbers that would show whether or not the United States was winning in Vietnam. Sure enough, the military responded with “body count” figures that showed the Viet Cong were all being killed many times over, but the United States lost the war.
I thought of Goodhart’s Law in connection with President Bush’s No Child Left Behind and President Obama’s Race to the Top programs. Teachers and schools are judged on the basis of test results. So the incentive is for teachers to improve, not learning, but the numbers by which learning supposedly is measured. Dishonest teachers cheat. Honest teachers have to take time away from teaching the material to teaching how to pass the test.
The aim is evidence-based policy. The result is policy-based evidence.
As Cory Doctorow explained on Boing Boing:
Once you start measuring GDP as a way of gauging social welfare, people will start to figure out ways to make GDP go up without improving social welfare (say, by swapping dirty financial derivatives). Once Google starts measuring inbound links as a way of evaluating the importance of web-pages, people will figure out how to increase the inbound links to unimportant pages (splogging, blogspam). And once you measure fat or calorie content as a proxy for the healthfulness of food, manufacturers will figure out how to decrease fat and calories without making the food more healthful (reducing fat by adding sugar, reducing calories by adding poisonous artificial sweeteners).
The prime example of Goodhart’s Law in action is Soviet economic planning. Factories were evaluated on the basis of measured output, irrespective of the usefulness of what was produced. Machinery factories were actually judged on the total weight of the machinery they produced. That is why there is no substitute for free markets and the workings of supply and demand. But large corporations often operate like mini-Soviet Unions until reality catches up with them.
W. Edwards Deming, who was possibly the world’s greatest exponent of using statistics to improvement business performance, objected to judging either managers or workers based on numerical goals. Understand and improve the process, and the numbers will improve, he said, but trying to improve the numbers without understanding the process is an exercise in futility.
Here are some links on Goodhart’s law.
The Importance of Goodhart’s Law
Articles by Charles A.E. Goodhart
Tags: Charles Goodhart, Economic Models, Economic Policy, Goodhart's Law, The Unofficial Rules
Phil Ebersole's Blog 
January 10, 2013 at 10:43 am |
This sounds like the open question objection well known to philosophers at least since Bishop Butler’s axiom, “a thing is what it is and not another thing.” If you try to measure goodness in terms of happiness as utilitarians still do, you are left with the open question: ok, this makes me happy, but does it make me good? Unfortunately, no one is paid to bring up the open question objection, but people are paid or otherwise rewarded for dreaming up new ways around Deming’s profound understanding of quality control. There are other sweeping examples from education. Since the 1960s quantitative measures have been used to allow students to grade teachers, the SAT and similar scores have been used to determine admissions, and the student’s GPA is used to determine graduate and job opportunities. All very neat and scientific. The only trouble is that the students quickly learned it was in their interest to give the teachers high ratings, and the professors found themselves looking better if they gave good grades, and the rich were able to afford coaches who taught them to get higher SATs without knowing more. There are no cynical bad guys here, just a cynical culture of people who are so eager to please their boss and their customers that they don’t take time to think.
LikeLike
January 12, 2013 at 8:04 am |
[…] Goodhart’s law: on not going by the numbers. […]
LikeLike
January 13, 2013 at 5:54 am |
Exactly what is happening here:-
http://business.asiaone.com/Business/News/Story/A1Story20120815-365526.html
On the ground, the story is very different.
So stats don’t always tell the full story.
LikeLike
January 13, 2013 at 6:23 am |
To be clear, I think good statistics, like good maps, are important tools of understanding. I quote statistics all the time on this web log. You can’t understand anything unless you have a way to compare it with something else. But the map is not the territory, and the problem comes when you think you are changing things on the ground when you are only changing them on the map.
LikeLike
April 3, 2013 at 11:08 am |
[…] of the practical application of this outside higher education (ambulance response time) here. Or, as Phil Ebersole says elegantly, “The aim is evidence-based policy. The result is policy-based […]
LikeLike
May 21, 2016 at 9:23 pm |
[…] A very good take on Goodhart’s Law and GDP can be found on Phil Ebersole’s blog: […]
LikeLike