Paste your Google Webmaster Tools verification code here

Statistical rigour and HS2

At the Treasury Select Committee, in relation to HS2, Professor Graham said that it was important that we get these numbers right and in a scientifically rigorous way.  Both he and Professor Overman attacked the work published by KPMG as not only insufficiently rigorous, but also on the grounds that the results are too optimistic.

I want to examine both these propositions.  I’d like to start by saying that in general I think that we tend to underestimate the benefits of transport investments, and this will include HS2.  One of the major reasons for this is that we have developed an analytical mechanism which is suited to ranking projects in a known economy and used it to evaluate projects in an uncertain future one. In an uncertain world, well into the future, it is essential that we are careful about what assumptions we are making about how far the future is different from the past.  Statistical rigour can obscure this.

The KPMG work has been attacked for being creative and taking a non-standard approach.  The Crossrail work on Wider Economic Benefits which I pioneered was also attacked for being non-standard and creative at the time – now I notice that it is considered reasonable that there is a productivity response to transport investment and Professors Graham and Overman support the idea.

The idea of ‘wider’ benefits is of course that they are an add on to ‘standard’ benefits. Standard benefits accrue to people and businesses in a known world as far as the project is concerned.  The forecasts of activity are independent of the transport system.  Wider benefits reflect the potential for relieving constraints on higher productivity activities arising from the transport system which mean that people do less productive work than they might otherwise be able to do.  The officially sanctioned estimates of the parameters to be used have been made by Professor Graham, who is understandably protective of the work that he has produced.  Standard benefits, which are largely measured in time savings, are equal to real output benefits IF AND ONLY IF firms and people undertake all projects which are even marginally profitable, AND that the values of time have been correctly measured.

While these are elegant theoretical propositions, it is clear that there are some fundamental uncertainties about this edifice.  What if the forecasts are wrong, or the values of time inappropriate, or not all investments can be made? More importantly, what if transport makes more difference to activity and productivity than we have so far identified?

Professors Overman and Graham stress the problem of ‘confounding’, which sounds like something out of Harry Potter and refers to mistakenly crediting (say) transport with (say) creating jobs, when actually it was down to (say) investment in skills.  A competent researcher looking at the historical data on jobs would want to attribute the change among these potential causes.  This historical accounting may however be only of partial help when considering the future, especially the long term future.  It will not tell us is whether the skills investment would have been useless without the transport, or indeed vice versa.  Moreover, most of the analysis in this area is done on data for a particular year, so does not ask about changes over time, although this is what it wants to capture.

In a similar vein, much is made of the problem of causation.  Does the economy generate the transport system or the other way round?  I must admit it seems hard to envisage exactly how the economy generates transport investments which are under the control of central government and slow and meticulous planning systems, but there will certainly be feedbacks as transport generates market access, economic success, congestion, and demands for further investment.

Both joint effects and feedbacks mean that using the historical data to estimate the right answers will be fraught with danger.

Of course the data itself is messy and imprecise.  Even knowing the number of jobs in an area is subject to errors and revisions, while other data on level of skills, or output is pretty woolly.  What is the output of the kinds of firms which operate in city centres, advertising agencies, or accountants, or computer programmers?  How do we value it?  This is a difficult problem and should not be underestimated.

A scientifically rigorous process cannot be based on inadequate data and on estimating impacts without enough alternative cases to see, for example, what would happen if we made the skills investment without transport and vice versa.  Statistical rigour is not a substitute for careful consideration of what the issues might be and what the statistics might miss, and which might be as important as that which is measured.  It would be better to be broadly right than precisely wrong.

Hence the Treasury Committee’s probing of the KPMG work for its robustness, reliability, and whether it is a forecast is understandable but misses the point.  No single piece of analysis will create a robust and reliable view.  But a number of different analyses will present ranges of potential outcomes which must then be compared and judged.

The KPMG work, as I understand it, took the DfT forecasts for growth in jobs in the UK economy and asked how they might be redistributed under a different transport pattern of availability.  They generated a relationship between levels of accessibility and output across the UK and used this to develop an average reaction, so that jobs became more productive as they move into more accessible locations.  (This is actually pretty similar to the view of wider economic benefits, but pulls together both these and the benefits to ‘standard’ users too.) KPMG’s work effectively takes the transport investment as necessary for the benefits of other investments (such as in skills) to be effective.  In other words, people’s skills will not earn the return the statistics suggest they should be able to unless there is additional transport.  This could well be correct, though a statistical decomposition ex post of the various impacts will not necessarily show this.

Their estimates do, however, miss out on any effect on the total number of jobs in the economy and any impact that increased freight might have; and this is a non-trivial shortcoming.  It takes a total growth forecast which stops growth only a few years after the opening of the line.

The DfT guidance on the evaluation of transport projects instructs that the key scenario is the standard one, based on time saved independently of the economy which is taken to be known.  This cannot be sensible for a project of this scale and game changing nature.  The guidance is not relevant here.  And statistical analysis is only relevant if it illuminates what kind and scale of game changing might be possible or relevant.  Professor Overman states there is a good case for rail, and also that we take far too long to make decisions.  In my view, this is partly the result of the attempt to find statistical rigour where none exists.  The long term effects of long term investment can be much more usefully informed by historical analysis and thinking about the kinds of opportunities that can be generated by such investment.

For example, regarding Crossrail, I judge that the final benefits of Crossrail will generate total output returns of around £80 billion on an investment of £15bn[1].  An investment in new railways of around £40bn should, based on analogy like this, generate £200bn, in new jobs, incomes, additional commuting to productive centres, taxes, dividends and so on.  This kind of broad, historically informed analysis is in line with the sort of results that KPMG have generated.  It can be improved, refined, redone, but it won’t actually produce a more rigorous estimate.

by Bridget Rosewell


[1] Not quite like with like but ball park

Leave a Reply


Subscribe to our newsletter

Subscribe to receive our monthly newsletter to keep in touch with what we're up to.

Contact Us

56-58 Putney High Street, London, SW15 1SF
Phone: 0208 878 6333
Visit Us On TwitterVisit Us On Linkedin