|
|
Signifying nothing? Jan 29th 2004 From The Economist print edition Too many economists misuse statistics FIGURES
lie, as everyone knows, and liars figure. That should make economists
especially suspect,
since they rely heavily on statistics to try and resolve a wide range of
controversies. For example,
does a rise in the minimum wage put people out of work? Are stockmarket returns predictable?
Do taxes influence whether a company pays dividends? In recent years, helped
by cheaper,
more powerful computers, and egged on by policy-makers anxious for their
views, economists
have analysed reams of statistics to answer such questions. Unfortunately,
their guidance
may be deeply flawed. Two
economists, Deirdre McCloskey of the University of Illinois, and Stephen Ziliak of Roosevelt University,
think their colleagues do a lousy job of making sense of figures, often
falling prey to elementary
errors. But their biggest gripe is that, blinded by statistical wizardry,
many economists fail
to think about the way in which the world really works. To
be fair, statistics can be deceptive, especially when explaining human
behaviour, which is necessarily
complicated, and to which iron laws do not apply. Moreover, even if a
relationship exists,
the wrong conclusions can be drawn. In medieval Holland, it was noted that
there was a correlation
between the number of storks living on the roof of a house and the number of
children born
within it. The relationship was so striking that, according to the rules of
maths that govern such
things, you could say with great confidence that the results were very
unlikely to be merely random.
Such a relationship is said to be “statistically significant”. But the Dutch
folklore of the time
that storks somehow increased human fertility was clearly wrong. Examples
of similar errors abound. W.S. Jevons,
an English economist of the mid-19th century, thought
that sunspots influenced crop yields. More recently and tragically, British
mothers have felt
the harsh effects of statistical abuse. An expert witness frequently called
to give evidence in the
trials of mothers accused of murdering their children argued that the odds of
more than one cot
death in a family were statistically so slim that three such deaths amounted
to murder. On this erroneous
evidence, hundreds of parents have been separated from their children and
many others
have been sent to prison. A
failure to separate statistical significance from plausible explanation is
all too common in economics,
often with harmful consequences. In a past paper*
Professors McCloskey and Ziliak attacked
other economists' over-reliance on statistical rather than economic
reasoning, and focused
on one case in particular. In
the 1980s, the American state of Illinois launched a programme to keep people
off the dole. Economists
asked whether its costs outweighed its benefits. One study estimated that the
programme
produced benefits that were more than four times as large as the costs.
Although this seemed
a good deal for taxpayers—and other tests seem to support this conclusion—the
authors of
the study rejected such a finding because they found that their estimate was
not statistically significant.
In other words, their results fell just short of 90% certainty—the usual,
though ad hoc, rule
of thumb for most economic work—of not being random. But
far from this being an unusual case, Ms McCloskey and Mr Ziliak
found that 70% of the papers published
during the 1980s in the American Economic Review (AER),
one of the most respected journals
of the dismal science, failed to distinguish between “economic” and
“statistical” significance.
They relied too much on numbers, and too little on economic reasoning. Increasingly insignificant The
two had hoped things might be getting better in recent years. The reverse
seems to be the case.
In their latest work**, Ms McCloskey and
Mr Ziliak looked at all the AER
articles in the 1990s,
and found that more than four-fifths of them are guilty of the same sin.
Indeed, so pervasive
is the cult of statistical significance, say the authors, that ever more
economists dispense
altogether with the awkward question of whether the patterns they uncover
have anything
meaningful to say about the real world. Examples
are legion, and can be found in the work of very distinguished economists. In
a widely quoted
study of the minimum wage two Princeton University professors, Alan Krueger
and David Card,
claimed to show that, contrary to what you might expect, a rise in minimum
wages caused less
unemployment, not more. Though their statistics looked compelling, professors
McCloskey and
Ziliak say, they seemed to indicate, at best, a
rise in employment so small as to be economically
insignificant. Moreover, the paper did not address why this surprising result
might be true
(although the authors have discussed that question elsewhere). Another
paper criticised by Ms McCloskey and Mr Ziliak is
one co-written by Gary Becker, a Nobel- winning
economist. This claims to show that addiction is rational, mainly on
the basis that people's response
to changes in price is statistically significant. This is interesting, but
does not really explain
much. The three authors offered little account of why people become
addicted—an odd life choice
for a rational person to make. Most
fundamentally, argue Ms McCloskey and Mr Ziliak,
the focus on statistical significance often means
that they fail to ask whether their findings matter. They look, in other
words, at things that are
statistically but not economically insignificant. Most people would prefer
their conclusions to be significant
in both senses. Failing that, economic significance is presumably the more
important. *
“The Standard Error of Regressions”. By Deirdre McCloskey and Stephen Ziliak. Journal of Economic Literature, March 1996 **”Size
Matters: The Standard Error of Regressions in The American Economic Review”.
(Forthcoming in the Journal of Socio- Economics) Copyright
© 2004 The Economist Newspaper and The Economist Group |