Posts Tagged ‘Charles Goodhart’

Goodhart’s law: on not going by the numbers

January 10, 2013

Charles Goodhart was an adviser to the Bank of England in 1975.  The advice he gave then has been summarized as Goodhart’s law, which has been summarized as follows:

All economic models break down when used for policy.

Charles Goodhart

Charles Goodhart

His version

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.’

Another short version

When a measure becomes a target, it ceases to be a good measure.

Defense Secretary Robert McNamara demanded a graph of numbers that would show whether or not the United States was winning in Vietnam.  Sure enough, the military responded with “body count” figures that showed the Viet Cong were all being killed many times over, but the United States lost the war.

I thought of Goodhart’s Law in connection with President Bush’s No Child Left Behind and President Obama’s Race to the Top programs.  Teachers and schools are judged on the basis of test results.  So the incentive is for teachers to improve, not learning, but the numbers by which learning supposedly is measured.  Dishonest teachers cheat.  Honest teachers have to take time away from teaching the material to teaching how to pass the test.

The aim is evidence-based policy.  The result is policy-based evidence.

As Cory Doctorow explained on Boing Boing:

Once you start measuring GDP as a way of gauging social welfare, people will start to figure out ways to make GDP go up without improving social welfare (say, by swapping dirty financial derivatives).  Once Google starts measuring inbound links as a way of evaluating the importance of web-pages, people will figure out how to increase the inbound links to unimportant pages (splogging, blogspam).  And once you measure fat or calorie content as a proxy for the healthfulness of food, manufacturers will figure out how to decrease fat and calories without making the food more healthful (reducing fat by adding sugar, reducing calories by adding poisonous artificial sweeteners).

The prime example of Goodhart’s Law in action is Soviet economic planning.  Factories were evaluated on the basis of measured output, irrespective of the usefulness of what was produced.  Machinery factories were actually judged on the total weight of the machinery they produced.  That is why there is no substitute for free markets and the workings of supply and demand.  But large corporations often operate like mini-Soviet Unions until reality catches up with them.

W. Edwards Deming, who was possibly the world’s greatest exponent of using statistics to improvement business performance, objected to judging either managers or workers based on numerical goals.  Understand and improve the process, and the numbers will improve, he said, but trying to improve the numbers without understanding the process is an exercise in futility.

(more…)