Reference class forecasting

Reference class tennis

The reference class problem, also known as reference class tennis, is the discussion of which reference class to use when forecasting a given situation.

Suppose someone were trying to predict how long it would take to write a psychology textbook. Reference class tennis would involve debating whether we should take the average of all books (closest to an outside view), just all textbooks, or just all psychology textbooks (closest to an inside view).

Practical use in policy and planning

Whereas Kahneman and Tversky developed the theories of reference class forecasting, Flyvbjerg and COWI (2004) developed the method for its practical use in policy and planning, which was published as an official Guidance Document in June 2004 by the UK Department for Transport. In policy and planning environments, reference class forecasting is strengthened when project teams measure variables that were previously treated as intangible, since quantifying these uncertainties creates more accurate distributions for costs, schedules and expected benefits. This measurable approach reduces optimism bias and supports more accountable planning decisions.

The first instance of reference class forecasting in practice is described in Flyvbjerg (2006). This forecast was part of a review of the Edinburgh Tram Line 2 business case, which was carried out in October 2004 by Ove Arup and Partners Scotland. At the time, the project was forecast to cost a total of £320 million, of which £64 million – or 25% – was allocated for contingency. Using the newly implemented reference class forecasting guidelines, Ove Arup and Partners Scotland calculated the 80th percentile value (i.e., 80% likelihood of staying within budget) for total capital costs to be £400 million, which equaled 57% contingency. Similarly, they calculated the 50th percentile value (i.e., 50% likelihood of staying within budget) to be £357 million, which equaled 40% contingency. The review further acknowledged that the reference class forecasts were likely to be too low because the guidelines recommended that the uplifts should be applied at the time of decision to build, which the project had not yet reached, and that the risks therefore would be substantially higher at this early business case stage. On this basis, the review concluded that the forecasted costs could have been underestimated. The Edinburgh Tram Line 2 opened three years late in May 2014 with a final outturn cost of £776 million, which equals £628 million in 2004-prices.

Since the Edinburgh forecast, reference class forecasting has been applied to numerous other projects in the UK, including the £15 (US$29) billion Crossrail project in London. After 2004, The Netherlands, Denmark, and Switzerland have also implemented various types of reference class forecasting.

Before this, in 2001 (updated in 2011), AACE International (the Association for the Advancement of Cost Engineering) included Estimate Validation as a distinct step in the recommended practice of Cost Estimating (Estimate Validation is equivalent to Reference class forecasting in that it calls for separate empirical-based evaluations to benchmark the base estimate):

The estimate should be benchmarked or validated against or compared to historical experience and/or past estimates of the enterprise and of competitive enterprises to check its appropriateness, competitiveness, and to identify improvement opportunities...Validation examines the estimate from a different perspective and using different metrics than are used in estimate preparation.

In the process industries (e.g., oil and gas, chemicals, mining, energy, etc. which tend to dominate AACE's membership), benchmarking (i.e., "outside view") of project cost estimates against the historical costs of completed projects of similar types, including probabilistic information, has a long history. A method combining reference class forecasting and competitive crowdsourcing, Human Forest, has also been used in the life sciences, to estimate the likelihood that vaccines and treatments will successfully progress through clinical trial phases.

In the book Anthropic Bias, philosopher Nick Bostrom described ways in which statistical reasoning about reference classes can be applied to answering scientific questions related to our existence in the universe, such as with the anthropic principle, the fine-tuned universe hypothesis and its possible explanations, the doomsday argument, the size of the universe and possibility of a multiverse, and thought experiments such as the sleeping beauty problem. Bostrom investigates how to reason when one suspects that evidence is biased by "observation selection effects", in other words, when the evidence presented has been pre-filtered by the condition that there was some appropriately positioned observer to "receive" the evidence.

Bostrom argues against the self-indication assumption (SIA), a term he uses to characterize some existing views, and introduces the self-sampling assumption (SSA): that you should think of yourself as if you were a random observer from a suitable reference class. He later refines SSA into using observer-moments instead of observers to address certain paradoxes in anthropic reasoning, formalized as the strong self-sampling assumption (SSSA): Each observer-moment should reason as if it were randomly selected from the class of all observer-moments in its reference class. These different assumptions are affected differently based on the choice of reference class. An application of the principle underlying SSSA (though this application is nowhere expressly articulated by Bostrom), is: If the minute in which you read this article is randomly selected from every minute in every human's lifespan, then (with 95% confidence) this event has occurred after the first 5% of human observer-moments. If the mean lifespan in the future is twice the historic mean lifespan, this implies 95% confidence that N < 10n (the average future human will account for twice the observer-moments of the average historic human). Therefore, the 95th percentile extinction-time estimate in this version is 4560 years.