Research and analysis

Optimism Bias and Contingency at Homes England, accessible version

Updated 12 February 2024

Applies to England

This document has been prepared for Homes England by:

Oxford Global Projects

John Eccles House

Robert Robinson Avenue

Oxford Science Park

Oxford

United Kingdom

OX4 4GP
https://www.oxfordglobalprojects.com/

Study team:

Mike Bartlett (Oxford Global Projects)

Dr Alexander Budzier (Oxford Global Projects)

Dr Sam Franzen (Oxford Global Projects)

Andreas Leed (Oxford Global Projects)

Contact:

[email protected]

0300 1234 500

gov.uk/homes-england

1. Foreword

Within our new Strategic Plan we have set ourselves the mission of driving regeneration and housing delivery, to create high-quality homes and thriving places. This will support greater social justice, the levelling up of communities across England and the creation of places people are proud to call home. A key part of delivering this ambition is ensuring that we invest public funds where they will deliver the most value for money. Rigorous economic appraisal, alongside evaluation, plays a central role in ensuring this can be achieved.   

An important assumption used to inform economic appraisal is optimism bias – the tendency for appraisers to be optimistically biased about key project parameters, including costs, project duration and benefits. The optimism bias adjustment can have a significant impact on the results of an economic appraisal, so it is vital that it is informed using the best available evidence. We have therefore commissioned new research focused on the application of a Reference Class Forecasting approach to better inform judgements of optimism bias in the context of Homes England’s interventions. This has focused specifically on cost estimation and, in addition to supporting more robust economic appraisal, the outputs from this work can also be used, alongside other methods, to inform financial appraisal and the assessment of contingency. 

The work undertaken through this research provides important new evidence for the Agency. However, we recognise that it is vital to continue to learn from our projects and to ensure the assumptions being used reflect the latest evidence. The background analysis used to inform this work will therefore be updated periodically to reflect the latest evidence and data from Homes England’s projects.  

This report is part of a broader programme of research we have been undertaking, working in close collaboration with colleagues in the Department for Levelling Up, Housing and Communities (DLUHC) and in consultation with HM Treasury, focused on strengthening Homes England’s ability to measure and assess the full social value. delivered through our housing and regeneration activities

I would like to thank Oxford Global Projects for leading this element of the research. I would also like to thank the many colleagues within Homes England and DLUHC for their input to the project. 

Andy Wallis   

Chief Economist, Homes England

2. Executive Summary

1: Homes England is the government’s housing and regeneration agency. We drive regeneration and housing delivery to create high-quality homes and thriving places. This will support greater social justice, the levelling up of communities across England and the creation of places people are proud to call home. We believe that affordable, quality homes in well-designed places are key to improving people’s lives. We make this happen by using our powers, expertise, land, capital and influence to both bring investment to communities and get more quality homes built.

2: The assessment of a project’s costs and benefits, in the form of economic appraisal, is an integral part of optimising the social value delivered through the use of public sector resources. Where possible, this should involve valuing and monetising all relevant costs and benefits. This goes beyond market effects and includes areas such as environmental and broader social impacts.

3: Substantial work has been undertaken across government to improve the approach to economic appraisal and the use of tools such as cost benefit analysis. The Department for Levelling Up, Housing and Communities (DLUHC) has recently updated its own DLUHC Appraisal Guide (2023). In conjunction with DLUHC’s work on updating its appraisal guidance, Homes England has implemented a comprehensive research programme, targeted at improving the way in which economic appraisal is used to accurately and consistency assess the full economic and social impacts of the Agency’s activities.

4: In 2022, Homes England commissioned Oxford Global Projects to apply the Reference Class Forecasting (RCF) approach in the context of projects and programmes supported by Homes England. The study provides information on expected cost and schedule overruns.

5: RCF is a technique used to inform cost and schedule forecasting and encourage proactive risk management. RCF does this by providing evidence-based contingency uplifts. The RCF approach is referenced by the Infrastructure Project Authority (IPA), applied by the Department for Transport^{[footnote 1]} and further afield RCF is endorsed by the American Planning Association and is recommended practice in Switzerland, Denmark, the Netherlands and Australia.

6: Reference class forecasts are based on the actual performance of past comparable projects, providing an “external” view which overcomes bias and other limitations of “internal” forecasting methods. By providing a range of outcomes based on the composition and maturity of the base cost and/or schedule estimates, RCF can be used in several ways.

7: For an economic appraisal, RCF can replace general optimism bias uplifts where forecast accuracy can be demonstrated. In this context RCF provides more precise uplifts based on the exact nature of the works, thus making it more accurate than generic uplifts. Further, because RCF provides a range of uplifts corresponding to different levels of probability (certainty of completing on time or budget and associated P-level^{[footnote 2]}) RCF can be used to stress test economic appraisal. For instance, the P-mean could be used in a central case, but a higher P-level could be used to determine if an economic appraisal remains positive under worse-case scenarios.

8: For a financial appraisal, RCF can be used alongside Quantitative Risk Assessments (QRA) and/or contingency recommendations provided by internal or external experts. If there is uncertainty over which contingency value to use, the difference between the contingencies recommended by each approach can inform the level of contingency to be held at portfolio or programme level. This ensures the financial case carries sufficient contingencies in the event of a project overrun.

9: In affordability driven budgets where the available contingency is limited by the budget envelope, RCF can be used to determine how much certainty the current level of contingency provides. This can be used to inform our understanding of the likelihood of the project delivering to cost and schedule. Ultimately, this enables a transparent risk appetite discussion and funders to own the risk.

10: To make the process of generating an RCF as easy as possible, an RCF Workbook has been developed and published alongside this report. This generates a project specific RCF based on the inputs from a standard Homes England Cost Schedule.

3. Reference Class Forecasting Background 

Why Reference Class Forecasting?

11: RCF is an established risk forecasting method that addresses the root causes of cost and schedule overrun by using historical project data as a predictor of the uncertainty and risk to future projects. These root causes, including optimism bias and strategic misrepresentation, can lead to the underestimation of project costs or schedule, which later results in overruns.

12: RCF removes biases by taking an “outside view” when estimating cost and schedule risk. It does this by using all the distributional information available on past project performance to generate a realistic range of potential outcomes (Figure 1 ).

Figure 1: Reference Class Forecasting: Summary

This diagram could not be made fully accessible. Please refer to the PDF version of this document.

13: According to leading academics such as Daniel Kahneman, considering the “outside view” alongside other forecasting methods such as QRAs results in more accurate estimates and higher-quality decisions. This may be considered the single most important piece of advice regarding how to increase accuracy in forecasting through improved methods^{[footnote 3]}. 

14: Independent research has also shown that RCF can outperform conventional forecasting and monitoring techniques^{[footnote 4]}.  

15: RCF has been used by the UK Department for Transport since 2004^{[footnote 5]} to estimate optimism bias as required by the HMT Green Book^{[footnote 6]}. The method has since been endorsed by the American Planning Association and is recommended practice in Switzerland, Denmark, the Netherlands, Ireland and Australia. The latest version of the HMT Green Book recognises that the use of reliable data from similar projects – as generated by an RCF – should replace the recommended optimism bias values.

What drives risk in infrastructure projects?

16: Most projects evolve as they progress from idea to reality. Changes may be due to uncertainty regarding the level of ambition, the exact location, technical standards, safety, environment, project interfaces and geotechnical conditions. Prices and quantities of project components are also subject to uncertainty, whilst the choice of procurement and contracting strategy can introduce changes to the risk profile of the project. This uncertainty reduces as the project matures and more becomes known. This is often portrayed as a maturity funnel (Figure 2).

Figure 2 Maturity Funnel

This diagram could not be made fully accessible. Please refer to the PDF version of this document.

17: ISO 31000 links uncertainty to risk by defining risk as the “effect of uncertainty on objectives”, with the effect being a positive or negative deviation from what is expected. More conventionally, risk is regarded as the adverse consequence of change.

18: As can be seen from many examples both domestically and internationally, the world of project preparation and implementation is a highly risky one where things happen unexpectedly, and outcomes rarely turn out as originally intended.

19: To mitigate uncertainty and risk in infrastructure projects, good risk management is required. As stated in the HMT “Orange Book”^{[footnote 7]}:

“In successful organisations, risk management enhances strategic planning and prioritisation, assists in achieving objectives and strengthens the ability to be agile to respond to the challenges faced. If we are serious about meeting objectives successfully, improving service delivery and achieving value for money, risk management must be an essential and integral part of planning and decision-making.”

What level of overruns are experienced by infrastructure projects?

20: Cost and schedule overruns are a problem in private and public sector projects and, based on international trends, things are not improving. Indeed cost overruns have remained high and constant for the 70-year period for which comparable data exists. Geography does not seem to matter either with all countries and continents, for which data are available, suffering from overruns^{[footnote 8]}.

21: Hence, when comparing the estimated cost and schedule at business case submission with the actual outturn at project completion, some degree of overrun risk generally exists. It is therefore vital that this risk is considered in project appraisal, programming, budget setting and project cost control.

22: Although the specific project risk is uncertain, the potential range of risk impacts can be known and should be reflected in the project management practices and forecasts at any given stage. Table 1 shows how projects actually deliver compared against expectations, based on international data.

Table 1: Typical RCF output data

	Solar power	Roads	Rail	Buildings	IT-led change	Dams	Olympics	Nuclear waste storage
Cost overrun	1%	16%	39%	62%	73%	75%	157%	238%
Frequency of cost overrun	4 of 10	6 of 10	7 of 10	7 of 10	4 of 10	7 of 10	10 of 10	9 of 10
Schedule overrun	2%	36%	32%	32%	43%	44%	0%	70%
Benefits overrun	n/a	-5%	-23%	-5%	-28%	-11%	n/a	-23%
Cost black swans	0%	4%	10%	20%	18%	23%	57%	43%

23: As an example relevant to the work of Homes England and other housing and regeneration organisations, looking at the buildings column in Table 1, we can see that: 

• 70% of projects had a cost overrun, with an average cost overrun of 62%
• The average schedule overrun was 32%
• The average benefits shortfall was 5%
• 20% of projects encountered “Black Swans” (a black swan is an event or occurrence that deviates beyond what is normally expected of a situation and is extremely difficult to predict. Black swan events are typically random and unexpected^{[footnote 9]}).

What are the causes and root causes of overruns? 

24: Frequently, funders, owner-operators and builders of projects tend to explain cost and schedule overruns in major projects as resulting from external events such as unforeseen ground conditions, project complexity, scope and design changes, bad weather, delays in site access and possession and delays in obtaining permits. See Cunningham 2017 for a review of studies on causes of cost and schedule overruns^{[footnote 10]}. 

25: However, these events are not the root cause of overruns. The root cause is that adverse events are systematically underestimated, and unforeseen risks can be ignored during project development and decision making. This means that even if known risks have been managed effectively, a project can still overrun because the underestimated risks will not be effectively managed or provided for. Cost overruns and schedule delays are therefore the materialised impact of underestimated risk.

Why is risk underestimated?

26: Research into the track record of past project performance (Flyvbjerg et al. 2004^{[footnote 11]}, Flyvbjerg 2011^{[footnote 12]}, 2014^{[footnote 13]}, 2017^{[footnote 14]}) shows that projects’ outturn cost and schedule are systematically higher than their planning estimates. Further, project overruns have stayed consistently high for at least 70 years.

27: This evidence means that cost and schedule overruns cannot simply be explained by bad luck or error. If this were true, underruns would be similar in scale to overruns and estimation accuracy would have improved over time. Instead, the data shows that there is a systematic bias towards underestimating cost and schedule risks.

28: Extensive research has shown that this bias is driven by two main processes: optimism bias and strategic misrepresentation.

29: Optimism bias is the natural human tendency to be overly optimistic about outcomes compared to what previous experience of carrying out a similar task would objectively suggest. Optimism bias therefore causes people to naturally underestimate the costs, completion times and risks of planned actions. People also overestimate the benefits of the same actions. Such errors of judgment are shared by experts and laypeople alike (Kahneman and Tversky, 1979^{[footnote 15]}).

30: Strategic misrepresentation is like optimism bias in that it causes people to underestimate risks. However, while optimism bias is unconscious, strategic misrepresentation is a conscious and deliberate under-reporting of potential cost and/or time outcomes or over-stating of benefits (Wachs 1989^{[footnote 16]}, Flyvbjerg 2005^{[footnote 17]}). This may be done for many reasons, but is usually a desire for the project to look more favourable than an objective assessment would suggest.

31: The impact of optimism bias and strategic misrepresentation, alongside other causes of planning bias, is known as the planning fallacy.

4. Reference Class Forecasting Theory

32: Section 3 explained that causes of cost overruns and schedule delays are often attributed to conventional explanations of material risk impacts such as unforeseen ground conditions, project complexity, bad weather and so forth. However, after the event, it is usually apparent that these risks could have, and should have been predicted. Bad weather and poor ground conditions are common problems. The reason these risks were unforeseen or underestimated at the estimating stage is due to the planning fallacy.

33: Without adequate provision, these “unforeseen” risk impacts cannot be absorbed and the project overruns its budget and/or timelines. However, project decision-makers can take the following steps to reduce bias in their project plans and proposals.

Take an outside view

34: The conventional “inside view” of project planning and evaluation results in optimistic estimates and plans. Planners and decision makers with an “inside view” focus on the constituents of the specific planned action rather than on the outcomes of similar actions that have already been completed, i.e., an “outside view”.

35: The industry standard of QRA has evolved to present range estimates with Monte Carlo simulations^{[footnote 18]}. However, these are based on the internal perspective of what risks are applicable and to what extent they should be quantified. As discussed earlier, these perspectives can be biased, and will only include known risks (‘known knowns’ and sometimes ‘known unknowns’). The analytical process also has limited ability to model complexity and compound risks. This often results in QRAs generating narrower ranges for outcomes than would be suggested based on past project performance. Further, Monte Carlo simulations are not a tool that automatically de-biases risk estimates and so will result in biased outputs if the inputs themselves are biased.

36: This is not to say that Monte Carlo analysis does not have value. The approach is important for quantifying the impact of individual risks and mitigation strategies, especially in the later planning stages when increased project definition allows for more certain specifications. But its outputs need to be considered in context of the overall project’s maturity (see Section 4).

37: The alternative “outside view” pools lessons from past projects. In its basic form, the outside view can be taken by comparing the project at hand to comparable past projects with a view to learn from them.

38: International research has shown that projects are typically weak in applying lessons learned from comparators and that this is linked to the perceived uniqueness of projects. When teams perceive their project to be unique, they can exclude the experience and knowledge gained from other projects because these are not considered relevant. In reality, unique projects are rare. Projects are typically specific to a location and a context, but they are rarely unique when looking at global experience and track record. Even the renovation of Westminster Palace has used RCF to predict cost and schedule risk.

39: Thus, as a first step, decision makers should challenge and evaluate the quality of estimates and plans by taking the outside view of their project.

Reference Class Forecasting

40: RCF is used to make explicit, empirically based adjustments to estimates that can reduce estimation biases. It also provides for ‘unknown-unknown’ and compound risks by accounting for all risks that have impacted past projects – something which is not possible in conventional forecasting.

41: To be robust, these adjustments should use data from past projects or similar projects elsewhere and be calibrated for the unique characteristics of the project at hand.

42: RCF follows three steps:

1. Select the reference class: Identify a sample of past, similar projects – typically a minimum of 20-30 projects is enough to get started, but the more projects the better.

2. Establish the probability distribution: Determine the risk of the variable in question based on the past projects – e.g., the cost or schedule or benefit fluctuation of these projects.

3. Adjust the current estimate: This is done through applying an RCF uplift to the base estimate stripped of any risk provision, or by asking whether the project at hand is more or less risky than projects in the reference class, resulting in an adjusted uplift.

Selecting the reference class

43: The reference class needs to be as broad as possible to avoid excluding valuable data points, while being as similar as necessary to the project being analysed. To avoid reintroducing bias in the selection of projects, statistical analysis is used to determine which projects are statistically comparable.

44: The maturity of the projects in the reference class should also be aligned to the maturity of the project being assessed. This ensures that we are comparing the project at hand to projects at a similar maturity baseline. For instance, if forecasting the cost risk of a project at Strategic Outline Case stage (SOC), data from projects that also estimated their costs at SOC should be used. This is because risk and uncertainty decrease as a project matures (see the Maturity Funnel – Figure 2). If we included projects that estimated their costs at Business Case stage, we may underestimate the risk at SOC.

45: As can be seen, availability and accessibility of good quality data on previous projects is key to the initial development of the reference class. Ongoing updates to the reference class with new data as and when additional projects are completed is also important as this strengthens the reference class by adding more relevant data points. Adding data from the organisation applying the RCF will also help to tailor the reference class to that organisation’s particular performance characteristics.

Establish the probability distribution

46: Next, the distribution of the data in question is analysed. For this, a cumulative distribution is constructed. In the case of cost overrun, the data are sorted from largest to smallest overrun and then the relative share of each data point in the sample is calculated and summed so that the distribution ranges from 0% - 100%. For example, if 25 projects are in a reference class each project has a 4% share – the project with the largest overrun represents 4%, the second highest overrun 8%, and so on.

47: Figure 3 depicts how the cumulative distribution curve of these data is then charted. For example, we can see that 50% of projects had an overrun of 60% or less, and 50% of projects had an overrun of 60% or more. At the higher end, we can see that 80% of projects had an overrun of 115% or less, and 20% had an overrun of 115% or more.

Figure 3: Cumulative probability distribution of overrun in the reference class (conceptual)

This diagram could not be made fully accessible. Please refer to the PDF version of this document.

Adjust the current estimate

48: The final step in RCF is to adjust the current estimate by applying the RCF uplift. It is important to note that the RCF uplift should be applied to the deterministic or base estimate, stripped of any risk and contingency. Therefore, if optimism bias or QRA risk has already been included in an estimate, this should be removed before applying the RCF uplift.

49: When reading the probability distribution (S-curve), the overrun becomes the uplift necessary to adjust the inside-view estimate. For example, if decision makers accept a 50% chance of overrun then the uplift corresponding to the 50th percentile (P50 - 60% in the above example) should be added to the base estimate.

50: The same principles apply when performing an RCF of project duration/schedule as they do for cost.

How is the RCF applied to business cases?

51: For an economic appraisal, RCF can replace general optimism bias uplifts where forecast accuracy can be demonstrated. RCF provides more precise uplifts based on the exact nature of the works, thus making it more relevant than generic uplifts. Further, because RCF provides a range of uplifts corresponding to different levels of probability (certainty of completing on time or budget and associated P-level) RCF can be used to stress test a business case. For instance, the P-mean could be used in a central economic case, but a higher P-level could be used to determine if a business case remains positive under worse-case scenarios.

52: For a financial appraisal, RCF can be used alongside QRA and/or contingency recommendations provided by internal or external experts. If there is uncertainty over which contingency value to use, the difference between the contingencies recommended by approaches can inform the level of contingency to be held at portfolio or programme level. This ensures the financial case carries sufficient contingencies in the event of a project overrun, allowing Homes England and other housing and regeneration organisations to drawdown the necessary reserve contingency rather than having to seek further authorisation.

53: In affordability driven budgets where the available contingency is limited by the budget envelope, RCF can be used to determine how much certainty the current level of contingency provides. This can be used to assess the risk of a project and the likelihood the project will deliver to cost and schedule. Ultimately, this enables a transparent risk appetite discussion and funders to own the risk.

54: Below we have provided an example of how RCF can be applied in practice ( Figure 4). Suppose Homes England had a project with a total base estimate of £100m. Traditional risk estimating processes recommended a contingency of 10% and so the total outturn is estimated to be £110m. When the RCF is run, it reveals that a 10% contingency only provides for 35% certainty of delivering at or below that estimate. This risk appetite is considered to be too high and so the project applies an RCF P50 to the financial case. The RCF P50 requires 25% contingency. The 25% uplift is applied to the base estimate stripped of any risk allocation and so it adds £25m to the £100m base estimate (not 25% on top of the original outturn estimate of £110m). Homes England may then decide to allocate £10m of the contingency to be managed by the project (the traditionally indicated contingency) and retain the remaining £15m to be managed at programme level (the difference between the traditional and RCF contingencies).

55: However, the project funders refuse the additional contingency recommended by the RCF because only £110m is available for the budget. In this scenario, the planners explain to the funders that this level of contingency (10%) only provides 35% certainty of completing on budget. The funders are then able to determine whether they are willing to accept and own this level of risk (65% chance of overrun), or whether to reconsider the budget envelope or project scope.

56: For the economic appraisal the planners decide to use the RCF instead of Green Book optimism bias uplifts. This is a standard civil engineering project and so requires an optimism bias uplift of 44% at Business Case stage according to the Green Book. However, the RCF shows the mean risk to be P60 which is 33%. In this case, the economic appraisal becomes more favourable. However, the planners want to check that the business case remains positive in the event of a worse-case scenario materialising. They decide that a P80 (51% overrun or uplift) represents a credible worse-case scenario and so they test the sensitivity of the business case to outturn costs of £151m. Under this scenario the business case remains positive (albeit reduced) which provides them the confidence needed to proceed.

Figure 4: Example application of RCF

This diagram could not be made fully accessible. Please refer to the PDF version of this document.

5. Guidance on applying RCF at Homes England

Which P-levels should be used?

57: There is no single or correct answer as to which P-level should be selected as it will depend on risk appetite and the circumstances in which the P-levels are being used, such as for financial or economic appraisal. The section below provides guidance on the approach taken by Homes England. This approach will be reviewed periodically.

Applying RCF in the Economic Case

58: From an economic appraisal perspective, the following outputs from the RCF tool should be used. These should be applied to a base estimate that excludes any contingency and any inflation allowance:

• Central estimate – The P-mean RCF should be used. This is a trimmed mean based on the P5 to P95 values and so excludes the impact of any outliers in the dataset
• Standard sensitivities – The P50 and P80 estimates should be presented as standard sensitivity tests unless alternatives are more appropriate. Where alternatives are used, the rationale for these should be explained within the business case document
• Contingency level – Where the proposed level of contingency in the financial case falls outside the range of the standard sensitivities, the economic case should be tested at the proposed contingency level so that the value for money implications of that level of cost can be interpreted
• Optimism Bias – A sensitivity test should be included in the economic appraisal at the level suggested by the standard Green Book adjustments
• Further sensitivity testing – Where proportionate, further sensitivity testing may be undertaken to, for example, test the switching value for the value for money category. This may be most relevant where the value for money category changes across the P50-P80 range and further understanding of the risk to value for money is required to inform the analysis

59: When adjusting these estimates to take account of inflation, Section 5 of this document provides the RCF associated with inflation adjustments. This allows the estimate to be built up as the base estimate (excluding any contingency and inflation allowance), the RCF risk allowance, the unanticipated inflation allowance (based on the inflation RCF) and the planned inflation allowance (applied to the base estimate, the RCF risk allowance and the unanticipated inflation allowance). Table 3 provides an example of this process. Once estimated, the effects of general inflation can be removed in the usual way for the economic case, as detailed within the HM Treasury Green Book.

Applying RCF in the Financial Case

60: The insight provided by the RCF analysis can also help to inform the project contingency requirement. Where proportionate and a suitably mature risk assessment is available, the RCF should be complemented by a QRA and expert judgement to ensure as deep an understanding of the projects risk and expected contingency requirement as possible is provided.

61: The following provides a brief explanation of the RCF and QRA methods to inform how to bring these together as complementary methods within the financial case.

62: RCF – The RCF approach takes an outside view of the project and is based on the observed performance of past similar projects. The approach is objective and so avoids issues with optimism bias (i.e. the systematic tendency to understate risk) and does not require a project level assessment of risk which may miss “unknown unknowns”. The approach will however only provide evidence on the range of outcomes of past projects and RCF uplifts assume that the planned project is representative of the typical project in the reference class. The RCF uplift therefore does not take account of insight into whether the project is more/less risky than the typical project in the RCF. The approach also does not focus on the information that will be needed to facilitate and incentivise effective risk management.

63: QRA – The QRA approach takes an inside view of the project and is based on the project team’s assessment of identified project risks, the scale of these risks and the associated probability of risks occurring. With the approach being based on the project team’s assessment of risk it brings the benefit of gaining buy in from the team and drives the ownership of risk and effective risk management. The approach is however subject to optimism bias (i.e. optimism over the scale and probability of identified risks) and risks may be excluded from the analysis (i.e. the unknown unknowns). It has also been observed that the approach can be limited in its ability to assess the interaction between risks.

64: According to Infrastructure and Project Authority guidance^{[footnote 19]}, at early planning stages when little is known about the project (SOC), RCF provides the best overall risk estimate. This is because the project is still too uncertain to produce a reliable bottom-up QRA estimate. As more is known about the project and risks become clearer at Business Case stage (BC), the QRA becomes more reliable, and the project may transition from a RCF approach to a QRA approach within the financial case. As the project matures to Point of Contract (PoC), the RCF is used as a benchmark to check the QRA is realistic and correct for any biases and unknowns that are present in the QRA.

65: See Figure 5 for a graphical representation of the IPA guidance, adapted from the IPA Project Routemap.

Figure 5: Infrastructure and Projects authority (2021). Project Routemap: Risk Management UK Module. London.

This diagram could not be made fully accessible. Please refer to the PDF version of this document.

66: In addition to the transition as a project matures, while both methods have been found to provide similar outcomes at the centre of the distribution, the RCF approach has been observed to be more representative at the extremes. Therefore, if there is interest in providing information on a high contingency scenario based on, e.g. a P80 or P90 estimate, it may be more appropriate to look to the RCF approach to inform this.

67: Ultimately, expert judgement will be required to interpret the outputs from the analysis undertaken and to determine how to use the evidence available to inform a view on the most appropriate contingency level for a specific project. The rationale for this judgement should be recorded and agreed through the decision-making process to ensure transparency and accountability.

68: To help to inform this judgement, the following should be considered:

1. Maturity of the project and its risk assessment. How mature is the project and the risk assessment? How confident can we be that the risk assessment is comprehensive (i.e. no missing risks or unknown unknowns)? How confident can we be that optimism bias has been minimised (e.g. the assessment of the scale and likelihood of risks is evidenced based on a representative set of past projects rather than a subjective judgement)?

2. Review differences between internal and external perspectives. Might differences be driven by the maturity of the project/risk assessment? Might there be gaps in the internal thinking? What risks may have been missed? Is there evidence to suggest the project is more/less risky than the average project that has the same spend category profile? (Note that objective evidence should be provided if this is found to be relevant).

3. Compare your project to its peer group and history. As an individual project, do you have a good chance of being in the best-case area of the curve? What is being done differently to maximise the likelihood of this? Are similar approaches being followed compared to past projects, if so then the project will likely, at best, be at the median level?

4. Output required from the analysis. What output from the analysis is required and what is this output being used to inform? Is clear information available on the broader portfolios risk appetite and/or the P-value that should be applied in different circumstances? Are we focused on a central estimate or information on potential contingency requirements at the extremes of the distribution?

5. Risk management. How can you incentivise teams and contractors to outperform the forecast? If there is a chance of delivering at the best end of the curve, how will you collectively achieve this? What evidence from the comparison can be used to determine whether areas of risk may not have been identified or that the risk assessment may have been subject to optimism bias? How might this information be used to ensure risk is controlled effectively?

Using RCF to inform programme and portfolio management

69: When looking at an individual project’s contingency requirement we will often be focused on a central estimate (e.g. the P-mean) or an alternative level linked to the organisations risk appetite (e.g. higher P-value to account for risk aversion in relation to a large project).

70: However financial management at the programme or portfolio level can be more complex and may require more thought to ensure the correct values are being used for the intended purpose. When managing a portfolio, consideration should be given to the extent to which risks between projects are common (i.e. if cost increases on one project, is it likely that they will also increase for the rest of the portfolio) or whether risks are unrelated and so may be pooled to some degree across the portfolio. It is therefore important for portfolio managers to consider the way the portfolio is constructed and the organisations risk appetite to provide clear guidance on the information needed from individual projects to facilitate effective management.

71: In addition to considering the make-up of the portfolio and overall risk appetite, consideration should be given to whether different outputs from the risk analysis are required for differing purposes. The following provide examples that may be relevant to your portfolio:

Tiered allocation of contingency.

• Suppliers – what level of contingency should be used to incentivise supplier performance?
• Project teams – incentivising project teams by allocating contingency based on their analysis of risk (e.g. ownership of QRA estimate) or at a lower P-value to incentivise risk management and prevent scope creep
• Portfolio – a higher tier of contingency held centrally with suitable controls around the draw down process

Purpose of estimate

• Managing a portfolio – As discussed above, the portfolio level of contingency may not necessarily equal the sum of individual project contingencies where risk can be pooled. The overall level of contingency held will also be influenced by risk appetite
• Seeking external approvals – It may be appropriate to seek approval to spend to a higher level of contingency for an individual project than the level of contingency proposed to be held against the project by the portfolio. This would, for example, reduce the need to seek reapproval where expected risk events occur and allow the portfolio to allocate contingency between projects. For example, if it were proposed to hold contingency at the equivalent of P-mean for each project in a portfolio, authority may be sought to spend a greater amount for individual projects (e.g. to P80) to allow for the portfolio to manage contingency between projects based on the expected distribution of risk without each risk event requiring additional approvals. There would however be a commitment to manage portfolio contingency to the sum of the P-mean levels and so additional contingency requirement from one project would need to be found from a lower than P-mean requirement on others

Continuous improvement

72: As explained previously, RCF uses external data on past project performance to forecast the risk of projects. As long as the data are tested for relevance, this is a valid and robust approach. However, the best predictor of an organisations performance is the performance of projects previously completed by that organisation. This is because each organisation may have specific traits that cause it to perform slightly differently to the projects in the wider reference class. That is why it is always recommended to capture internal data on project performance.

6. Inflation

Treatment of inflation in cost risk RCFs

73: RCF cost risk excludes the effect of inflation. This is because the projects included in the reference class are normalised to the same cost year in terms of their estimates and actual outturn, i.e., they use real-term cost estimates and real-term outturns to calculate overruns. Cost overrun in real terms is calculated as:

Actual cost in real terms/Estimated cost in real terms

74: This ensures that the cost risk calculation only reflects the impacts of cost overrun due to risks materialising, and not price increases due to inflation.

Estimating inflation and providing for unanticipated inflation 

75: In some cases, projects will have to produce estimates that include inflation and therefore additional consideration needs to be given to any optimism bias included in inflation forecasts. 

76: Typically, Homes England uses standard indices from recognised forecasting bodies to estimate their project inflation e.g., the Tender Price Index from the Building Cost Information Service (BCIS). However, forecasting inflation is uncertain. Therefore, these indices sometimes underestimate future inflation. This is problematic because it may leave the project with insufficient inflation allocation and so result in the project overrunning its estimate – not necessarily because the project went wrong, but because the inflation forecast was optimistic. 

77: To help to avoid this, further contingency for inflation can be applied, as you would for cost risk. This provides for unanticipated inflation. 

78: Table 2 provides an RCF for unanticipated inflation. This is based on 116 projects where both nominal and real-term cost overruns were available. Unanticipated inflation is calculated as the difference between nominal and real-term overruns for these projects:

Unanticipated inflation = Cost overrun in nominal terms - Cost overrun in real terms

79: As presented in Table 2, 25% of projects correctly estimated their inflation or provided for more inflation than was needed. 50% of projects exceeded their inflation allocation by 4% or less, and 80% exceeded their inflation allocation by 10% or less. Therefore, if you wanted to be 50% certain your inflation allocation will be sufficient you should apply a 4% uplift for unanticipated inflation. If you want to be 80% certain your inflation allocation is sufficient you should add 10% uplift for unanticipated inflation. Uplifts should be applied to the base estimate excluding risk and planned inflation allocation.

80: The unanticipated inflation RCF and this approach to inflation treatment applies analysis undertaken for the Department for Transport and follows the methodology suggested in Annex D of the Transport Analysis Guidance document “Updating the evidence behind the optimism bias uplifts for transport appraisals”^{[footnote 20]}.

Table 2: RCF for Unanticipated Inflation

Cumulative Percentile or Certainty that Inflation Allocation is Sufficient	Unanticipated Inflation (N=116) or Uplift Required for Desired Level of Certainty
5%	-4%
10%	-2%
15%	-2%
20%	0%
25%	0%
30%	2%
35%	2%
40%	3%
45%	3%
50%	4%
55%	4%
60%	5%
65%	6%
70%	7%
75%	8%
80%	10%
85%	11%
90%	12%
95%	15%
Mean	4%

81: It may be argued that current inflation is unprecedented and so these forecasts based on historical data may be insufficient. However, the projects in the forecast cover a period of 20 years, during which time there have been periods where inflation has been similar to current levels. Nevertheless, it is important to note that the forecast essentially averages out inflation over the 20-year period. Therefore, in periods of unusually high inflation, higher P-levels could be used to minimise unanticipated inflation. In periods of unusually low inflation, lower P-levels can be used.

Adding contingency for unanticipated inflation to your estimate

82: The uplift for unanticipated inflation should be applied to the base estimate excluding risk and inflation. The value of this uplift should then be added to the estimate including risk. Finally, the planned inflation allocation should be added to the base estimate including risk and unanticipated inflation. A worked example is provided below in Table 3.

Table 3: Example of cost estimate build-up including risk, unanticipated inflation, and planned inflation allocation.

Estimate components	Uplift	Value	Total
Base Estimate	NA	100
RCF Cost Risk applied to Base Estimate (example)	10%	10	110
Uplift for Unanticipated Inflation applied to Base Estimate (P50 example)	4%	4	114
Planned inflation allocation applied to Base Estimate + Risk + Unanticipated Inflation (example)	5%	5.7	119.7

7. The RCF Workbook

83: To make the process of generating an RCF as easy as possible, the RCF Workbook has been developed. This integrates with Homes England cost estimating processes by automatically generating a project-specific RCF based on the inputs from a standard Homes England Cost Schedule.

How is the RCF Workbook constructed?

84: A work breakdown structure was developed to include all the main types of works considered by Homes England. Cost and schedule overrun RCFs were then developed and assigned to each of the 139 cost categories and headings. For each RCF, different risk profiles are provided for each of the common Homes England business case stages. The RCFs for each of the included works are then blended according to the proportion of costs in each heading to produce a single overall risk profile tailored to the project.

What data does the RCF Workbook use?

85: The RCFs in the Workbook are based on data from OGP’s database. OGP hold the largest academic-quality dataset on project performance in the world. The use of the OGP data provides confidence that the forecasts provided are based on the most robust database available. The data covers more than 16,000 projects worth 3 trillion USD across all key sectors. OGP hold cost and schedule estimates and outturn for whole projects, as well as the major works associated with construction. OGP frequently employ these data to produce benchmarks by breaking down outturn cost and duration by key project parameters. For example, the OGP data are used by the U.K. Infrastructure and Projects Authority for their Benchmarking Hub, and HMT for updating the Greenbook Guidance.

86 Relevant data for Homes England has been gathered from multiple, national and international data sources. OGP thoroughly assessed the appropriateness of projects included in the Homes England RCFs, including through statistical testing of risk profiles. For instance, can Portuguese roads be used to forecast the risk of UK roads? The answer is no because they are statistically different, so Portuguese roads are excluded.

87: As much as possible, OGP sought to use RCFs that have been adopted and applied by Homes England partners e.g., the Department for Transport, the Department for Levelling Up, Housing, and Communities, and HMT. This ensures that the risk profiles being used for the same types of works are aligned, wherever appropriate.

88: Summary information on key works RCFs is presented in Table 4.

Table 4: Summary information on RCFs for key Homes England works.

Cost RCFs at Point of Contract Stage

RCF	Relevant HE Works	Project characteristics	Sample size	P50	P80	Trimmed Mean	Wider use
Land and Property Acquisition	Acquisition costs and associated cost/fees	Purchasing and acquiring land for government project purposes	48	-4%	11%	-13%	Transport Infrastructure Ireland
Standard Roads	General Unspecified Roads, Urban/Rural Standard Roads, Spine Roads, Bus Lanes	Mix of minor and major roads, including complex junctions, trunk roads, motorways	202	18%	37%	21%	Department for Transport – Standard TAG uplifts
Low Risk Roads	Footpaths, Cycle Paths, Parking	Deflation factor applied to Standard Roads RCF	202	16%	33%	19%	See above
Complex Risk Roads	Urban/Rural Complex Roads, Roundabouts, Junctions	Inflation factor applied to Standard Roads RCF – calibrated against Motorways RCF so that Complex Roads are higher risk than Standard but not as high as Motorways	202	22%	44%	25%	See above
Uncomplex Buildings (low risk)	Standard construction, inc. most HE buildings	Filter of OGP database to include uncomplex buildings such as: residential, schools, community buildings, GP practices, car Parks, small sports centres	64	4%	32%	22%	Department of Levelling Up, Housing, and Communities
Buildings (med risk)	Custom build properties	Buildings, including some more complex (Museums/Offices/Stadiums), with a duration of 0-5 years	42	9%	69%	28%	Treasury Greenbook
Mixed Utilities	Unspecified Utilities	Blend of utilities - Gas (1/3), Water (1/3), Electric (1/6), & Telecoms (1/6)	403	10%	32%	15%	DLUHC
Enabling Works	External Works, Simple Remediation, Surveys	Temporary works on railways	149	13%	84%	46%	East West Rail and others

How to use the RCF Workbook?

89: The RCF Workbook provides a tool to construct RCF estimates for a project. For cost, these estimates are based on the individual components of the project with, for example, separate reference classes being used for the purchase of land and spend on utilities. The RCF Workbook is available alongside this document. The remainder of this section provides guidance on completing the workbook and locating the outputs of interest.

90: Guidance on how to input the required information can be found on the Cover Sheet tab within the workbook and further notes on definitions are provided throughout the Input tab.

91: Key inputs include:

• Complete those cells shaded yellow at the top of the Input tab. This provides an important record of when the analysis was completed and information on project maturity which determines the reference class being used. Annex A provides a detailed project maturity matrix that can be used where a project is expected to differ in maturity compared to the average project at the given business case stage
• Input cost information into Column E. Rows 8 through to 172 provide detailed categories of spend expected to be involved in a Homes England project. Several categories of spend allow for either a detailed or higher-level breakdown. The aim should be to provide as detailed breakdown as is possible at that point in time
• Where Modern Methods of Construction (MMC) are being used to deliver an area of spend, the drop-down box in Column G should be marked Yes. The total value of work delivered using MMC should also be entered in Cell E53
• Where an area of spend should be excluded from the analysis, the dropdown box in Column I should be marked Exclude. This allows for certain elements to be excluded, for example due to contracts for that area of spend already having been agreed
• Rows 185 to 191 allow for an assessment of schedule

92: The Outputs tab provides a broad range of information on the output from the RCF analysis. Key outputs include:

• Tables detailing the components of cost that have been entered. Both for the project as a whole and taking account of any exclusions entered in the Input tab
• Table detailing the outputs from the RCF. The rows beginning J20, J26 and J31 provide the P50, P80 and P-mean estimates respectively. Column L provides values based on all cost categories, Column M provides adjusted values where MMC is being used to deliver elements of cost and Column N filters for those cost items marked for exclusion
• The table starting in cell J36 provides detail on the certainty given to the contingency estimate that has been entered on the Input tab. i.e. if this reports a value of 55%, that is saying the contingency held is equivalent to the P55 RCF estimate
• Rows 60 and below provide a similar set of information on project schedule

Annex A: Project Maturity Matrix

93: The table below details the maturity assessment that can be used to inform which RCF to apply. The top rows detail the Homes England Business Case stages and the dataset that has been used to estimate the RCF for those as standard. This is based on a standard assessment of project maturity at any given stage.

94: Where project maturity is expected to differ from this standard approach, maturity can be assessed using the matrix below. If this assessment suggests that a project is more/less mature than is standard for its Business Case stage, then the decision can be taken to apply an alternative RCF by changing the Business Case stage within the input tab to the workbook. This decision should be recorded and justified within the business case and any decision papers.

Homes England Stage	Project Concept/SOC	Business Case	Point of Contract/Delivery Phase
Dataset Used	SOBC	OBC	FBC
Business case development	Preliminary business case is authorised for detailed investigation. The business case appears to meet business needs, be affordable, achievable, with appropriate options suggested and has potential to achieve value for money.	A second iteration of the Business Case illustrating that there is a preferred option for the project is authorised.	A full business case confirming all material scope decisions and supported by relevant information from potential suppliers and/or delivery partners is authorised.
Funding	Funding is secured for the development phases.	Funding is secured in principle for the whole project.	Funding is fully assured with adequate contingency for the entirety of the project and potential risks.
Identification of business goals	The project is assessed as likely to deliver business goals.	The business goals delivered by the project are objectively quantified and confirmed.	The project is confirmed to deliver against quantifiable business goals.
Definition of outcomes/ benefits	The full scale, intended outcomes, timescales and impact of relevant external issues have been considered.  Desired benefits have been clearly identified at a high level, together with measures of success and a measurement approach.	The intended outcomes, timescales and impact of external and internal issues have been defined. Desired benefits have been objectively quantified. Objectives and desired outputs of the project are aligned with the programme to which it contributes.	The business has prepared for the development, implementation, transition and operation of new services/facilities. The objectives and desired outputs of the project are still aligned with the programme to which it contributes.
Project is benefiting from expert advice	Appropriate expert advice is being utilised to identify and/or analyse potential options.	The Business Case is supported by relevant expert advice with no significant dissent to the preferred option.	The Full Business Case is supported by relevant expert advice with no significant dissent to the project receiving full funding.
Definition of scope	The scope and requirements are realistic, clear and unambiguous with designs at RIBA stage 1 or equivalent.	The scope is well defined (RIBA Stage 2 or equivalent) with option selection clearly rationalised.	The scope is predominantly signed off (RIBA Stage 3 or equivalent) with no significant aspects still to develop.
Feasibility studies/ surveys	Feasibility studies have been completed satisfactorily and there is a preferred way forward.	Essential surveys have been completed and confirm feasibility of option.	All initial surveys have been completed and confirm scope.
Supply chain involvement	The supply chain’s likely interest and ability to deliver the objectives has been considered and developed in dialogue with the market as appropriate.	The supply chain capability to deliver the project is fully understood and there will be an adequate competitive response (if desired) from the market to the requirements. A contract management strategy is defined.	Recommended contract strategies are assured to deliver the specified outputs/outcomes on time, within budget and provide value for money. The Business Case and Benefits Plan are supported by relevant information from potential suppliers and/or delivery partners.
Project management established	The project is adequately managed with a competent and adequately resourced team.	Project controls and organisation are defined, financial controls are in place and the resources are available.	Management controls are in place to manage the project through to completion.  The development and implementation plans of both the client and the supplier or partner are sound and achievable.
Recognition of risks	Major risks have been identified and outline risk management plans have been developed.	Risks for all aspects of the project have been considered, accountability has been outlined and management plans are in place for currently active risks.	Plans for risk management, issue management (business and technical) are in place and shared with suppliers and/or delivery partners.
Delivery plans	There are plans for the next stage and an indicative delivery strategy to completion.	The project’s plan through to completion is appropriately detailed and realistic. The delivery strategy is robust and appropriate  Quality procedures are identified and applied consistently.	The technical implications, such as ‘buildability’ for construction projects; and for IT-enabled projects, information assurance and security and the impact of relevant procurement/ design etc frameworks have been addressed.
Approvals	There is internal and external authority, if required, and general support amongst essential, authorising stakeholders for the project. Planning assumptions are appropriate.	All significant approvals on track for satisfactory resolution. For construction projects, compliance with health and safety and sustainability requirements.	There is continuing support for the project with all major approvals secured.  The project can facilitate good client/supplier relationships in accordance with government initiatives. All necessary statutory and procedural requirements were followed throughout the procurement/evaluation process.
Source: Adapted from Oxford Global Projects, 2023

1. Department for Transport & Oxford Global Projects (2020). Updating the evidence behind the optimism bias uplifts for transport appraisals: 2020 data update to the 2004 Guidance Document Procedures for Dealing with Optimism Bias in Transport Planning. Department for Transport. 2020. ↩

2. The P-level refers to the percentile values taken from a distribution. For instance, the P50 refers to the median and the P80 is the 80th percentile. In the case of cost and the use of RCF, the P80 value therefore refers to the value where 80% of projects in the reference class had a cost overrun of less than this value. ↩

3. Kahneman (2011). Thinking, Fast and Slow. Penguin. 2012 ↩

4. RCF used at Sydney Water Corporation on 11 infrastructure projects showed significantly increased likelihood of completing under budget (Napier & Liu 2008); Hybrid method including RCF used at Australian State Road and Traffic Authority on 44 projects showed increased forecast accuracy (Liu, Wehbe & Siscovic 2010); Bridge construction forecast based on Bayesian updating and RCF produced more accurate forecasts (Kim & Reinschmidt 2011); RCF integrated in a Bayesian forecast of healthcare cost in 8 car manufacturing plants produced more accurate forecasts (Bordley 2014); Study of 56 construction projects shows that RCF outperforms conventional techniques, i.e. bottom-up estimation EVM and Monte Carlo simulations (Batselier & Vanhoucke 2016); Application of RCF to Bujagali hydropower dam project increased accuracy of the cost-benefit analysis (Awojobi & Jenkins 2016); Study of 399 political forecasters shows that those trained and using RCF, taking different perspectives, and post-mortem analyses produced more accurate forecasts (Chang, Chen, Mellers & Tetlock 2016); Integrating RCF into EVM on 23 construction projects produces more accurate predictions of schedule performance (Batselier & Vanhoucke 2017) ↩

5. Department for Transport & Oxford Global Projects (2020). Updating the evidence behind the optimism bias uplifts for transport appraisals: 2020 data update to the 2004 Guidance Document Procedures for Dealing with Optimism Bias in Transport Planning. Department for Transport. 2020. ↩

6. HMT (2022). The Green Book (2022). Government Finance Function. Updated 18 November 2022. ↩

7. HMT (2013). The Orange Book (2013). Government Finance Function. Updated 23 August 2021. ↩

8. Flyvbjerg, B., Holm, M.S. and Buhl, S.L. (2002) Underestimating Costs in Public Works Projects: Error or Lie? Journal of the American Planning Association, 68, 279-295. ↩

9. Taleb (2007). The Black Swan: The Impact of the Highly Improbable. Penguin 2008. ↩

10. Cunningham (2017). What Causes Cost Overruns on Building Projects? - An Overview. Review. School of Surveying and Construction Management. ↩

11. Flyvbjerg, B., Holm, M.K.S., Buhl, S.L., (2004). What Causes Cost Overrun in Transport Infrastructure Projects?. Transport Reviews, vol. 24, no. 1, January, pp. 3-18 ↩

12. Flyvbjerg (2011). Over Budget, Over Time, Over and Over Again: Managing Major Projects. Peter W. G. Morris, Jeffrey K. Pinto, and Jonas Söderlund, eds., The Oxford Handbook of Project Management, Oxford: Oxford University Press, pp. 321-344 ↩

13. Flyvbjerg (2014). From Nobel Prize to Project Management: Getting Risks Right. In book: Megaproject Planning and Management: Essential Readings, vol. 1 (pp.457-467) Chapter: From Nobel Prize to Project Management: Getting Risks Right ↩

14. Flyvbjerg (2017). Introduction: The Iron Law of Megaproject Management (April 2017). Bent Flyvbjerg, 2017, ed., The Oxford Handbook of Megaproject Management, Oxford University Press, Chapter 1, pp. 1-18. ↩

15. Kahneman, D. and Tversky, A. (1979) Prospect Theory: An Analysis of Decision under Risk. Econometrica: Journal of the Econometric Society, 47, 263-291. http://dx.doi.org/10.2307/1914185 ↩

16. Wachs, M. (1989) When planners lie with numbers, Journal of the American Planning Association, 55(4),pp. 476–479 ↩

17. Flyvbjerg (2005). Design by Deception: The Politics of Megaproject Approval. Harvard Design Magazine, no. 22, Spring/Summer, pp. 50-59 ↩

18. Monte Carlo analysis is a simulation-based risk modelling technique that produces expected values and confidence intervals as a result of many simulations that model the collective impact of a number of uncertainties. ↩

19. Infrastructure and Projects authority (2021). Project Routemap: Risk Management UK Module. London. ↩

20. Oxford Global Projects (2021). Updating the evidence behind the optimism bias uplifts for transport appraisals. Department for Transport. London. https://www.gov.uk/government/publications/tag-updated-evidence-for-optimism-bias-uplifts ↩