The story of the week for many people was the new alcohol guidelines issued by the UK’s chief medical officers. In 1995, the recommended weekly upper limit for men was set at 21 units, or around eight pints. This has now been slashed to only 14 units.
We might imagine that this drastic reduction is based upon some important advance in medical knowledge. But our cultural cousins in Ireland, Australia and New Zealand do not seem to have noticed it. They still have recommended limits very close to our 1995 one. Large swathes of America retain a puritanical mistrust of alcohol, and even pensioners are required to certify their age in many states before they can order a drink. But the “health limit” for men in the United States is 25 units. In Spain it is 35.
The methodology of science may seem an abstract subject, but it has important practical implications. The point in question is: can your conclusions be replicated by other scientists? Medical officers in the UK have used scientific evidence to pronounce about “safe” alcohol limits. Their Spanish counterparts have done the same, and have come up with a number two and a half times as large. The conclusions are completely different in the two cases.
Replicability is currently a hot topic in science. For example, there is a serious crisis in psychology. It seems that most of the conclusions they draw from their experiments cannot be reproduced when the tests are repeated. We have all read media features under the headline “six steps to happiness”, or some such compelling title, based on academic psychology. But we need to take them even less seriously in future.
Science and Nature are the top two scientific journals in the world. Last August, Science had an article which attempted to reproduce the results of 100 experiments published in leading psychology journals. The original teams collaborated with the replicators, a fact which should enhance the rate of replicability. In fact, only 36 per cent of the attempted replications led to results which were statistically significant. Further, the average size of the effects found in the new studies was only half that reported in the original studies. The lead author, Brian Nosek, commenting on the paper in Nature, said that there is no way of knowing whether any individual paper is true or false from this work!
Economics has made progress in facing up to this crucial issue. Can your result be repeated by someone else? Many leading journals now insist that the data sets and even the code used to generate findings are posted on line. But there is still a long way to go to get economists to take replication as seriously as do physicists. A few years ago, my own company created a competition, with a decent prize, for the best paper on replicating an article already published. In the first year we got five entrants, the second just two, and in the third only one. Economics must avoid the current pitfalls in medicine and psychology.
Paul Ormerod
As Published in CITY AM on Wednesday 13th January 2015
Image: Pour by jenny downing licensed under CC BY 2.0
