Research and analysis

Economic evaluation 12 month outcomes report: Health-led Employment Trial

Updated 8 April 2024

Applies to England, Scotland and Wales

First draft March 2022

DWP research report no.1036 A report of research carried out by the Institute for Employment Studies on behalf of the Department for Work and Pensions.

Crown copyright 2023. You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence. To view this licence, visit national archives

or write to the Information Policy Team

The National Archives
Kew
London
TW9 4DU

or email [email protected]

This document/publication is also available on our website Department for Work and Pensions

If you would like to know more about Department for Work and Pensions (DWP) research, email [email protected]

First published July 2023

ISBN 978-1-78659-548-5

Views expressed in this report are not necessarily those of the Department for Work and Pensions or any other government department.

Executive summary

This report provides detailed insights into the economic evaluation for the Health-led Employment Trials (HLTs). It assesses both costs and benefits to the exchequer, known as the financial case, and wider costs and benefits to society, known as the economic case. Costs and benefits were estimated under optimistic and pessimistic scenarios, as well as a reference scenario which fell between these two extremes.

The analysis demonstrated:

• There was local variation in how the Individual Placement and Support (IPS) services operated. Differences between the sites in the relative weight attached to achieving either health or employment outcomes had an important bearing on the ratio of benefits to costs.

• The positive impact of the IPS services on employment for the Sheffield City Region in-work trial group (SCR IW) and the West Midlands Combined Authority (WMCA) trial group resulted in some modest financial benefits. However, overall, the lack of earnings effects from the IPS services across WMCA and SCR, including for the in-work (IW) and out-of-work (OOW) groups, meant that benefits to the exchequer were insufficient to offset the costs of providing the IPS services.

• As the IPS service had no identifiable impact on health in WMCA, the benefits to costs ratio (BCR) for the economic case was zero or close to zero even under the most optimistic assumptions.

• The economic case for the IPS service was stronger in SCR than in WMCA, mainly due to the greater positive impact that it had on health-related quality of life for the SCR IW and SCR OOW trial groups. Under the reference scenario this resulted in £2.32 of benefits for every £1.00 of expenditure on the service for the SCR IW group and £2.02 of benefits for every £1.00 spent on the SCR OOW group.

• Although the economic BCR was greater for the SCR IW group than the SCR OOW group, differences in response rates between the treatment and control groups for the SCR IW group may affect the reliability of this finding.

• When the OOW trial groups were pooled (combining SCR OOW and WMCA), the economic benefits from the IPS services outweighed the costs under all but the most pessimistic assumptions. Under the reference scenario, every £1.00 spent on the IPS service resulted in £1.22 of benefits.

• The main analysis suggested that, for the SCR IW and the SCR OOW trial groups, the intervention did provide value for money when considering the costs and benefits to society. This was the case even under the most pessimistic assumptions explored in the report. However, a Monte Carlo probabilistic sensitivity analysis showed that if the study was repeated, there is a possibility that the economic benefits of the IPS service would not outweigh the costs in SCR even under the most optimistic scenario. As a result, the reported BCR should be treated with a high degree of caution.

• As the impact study was only able to consider impacts 12 months after randomisation for the full cohort of recruits in the treatment group across all outcome measures, there was a degree of uncertainty about how impacts might change over time and how this might affect the BCR.

• Gaining access to further extracts of administrative data would make it possible to explore impacts over a longer time period. This would reduce some of the uncertainty surrounding the BCR and increase the feasibility of estimating BCRs over more than 3 years. This would be likely to strengthen the economic case for the IPS service.

The final report series for the trials covers:

• Synthesis report: a high-level, strategic assessment of the achievements of the trial, drawing together the range of analyses from the evaluation.

• 4-month outcomes report covering: an analysis of implementation, a descriptive analysis of the survey findings 4 months post-randomisation, and an assessment of impact at 4 months following randomisation.

• 12-month survey report providing a descriptive analysis of the final survey, based on the theory of change for those in the treatment group.

• Context mechanism outcome (CMO) report, reporting evidence on outcomes from the trials and relating these to its theories of change.

• 12-month impact report covering the net effect on employment, health and wellbeing resulting from the trials 12 months after randomisation drawing on administrative and survey data.

• Economic evaluation report exploring the costs and benefits arising from trial delivery, drawing on the administrative and survey data.

• The pandemic and the trial – an analysis of how the trial outcomes may have been affected by the onset of COVID-19.

Acknowledgements

The authors gratefully acknowledge the contribution of all parties noted in the synthesis report, without whom the research presented here would not have been possible. We would also like to thank Becca Gooch and Jess Elmore from the Learning and Work Institute for their assistance in collecting data from providers. We are grateful to colleagues and former colleagues at the Institute for Employment Studies for their helpful contributions to data collection and reporting, including Dafni Papoutsaki, James Cockett, Rosie Gloster, Becci Newton and Rebecca Duffy. Any errors or omissions remain the responsibilities of the authors.

Author’s credits

Charlotte Edney is a Research Fellow at the Institute for Employment Studies (IES). She completed her PhD in Economics at Lancaster University where she gained experience in using a wide range of quantitative methods. In her work she has used various UK household surveys, cohort studies and administrative data.

Helen Gray is a Principal Research Economist at IES. She has particular expertise in the causal identification of impact using quantitative methods and linked administrative datasets. As well as leading the economic evaluation of the HLTs, she was responsible for the manipulation of DWP and HMRC data for the impact evaluation.

Dan Muir is a Research Officer at IES. He has supported the evaluation’s 12-month impact analysis as well as the economic evaluation. Dan joined IES in September 2021 after completing his MSc Economics studies at the University of Bristol. He has experience working with a range of large datasets and quantitative analysis in various policy-related projects.

Becci Newton is Director of Public Policy and Research at IES and specialises in research on unemployment, inactivity, health, skills and labour market transitions. Becci has managed the evaluation since its design and contributed to the process evaluation. She has led multiple evaluations for DWP including of the 2015 Employment and Support Allowance (ESA) Reform Trials and the Work Programme.

Rosie Gloster is a Senior Research Fellow at IES. She supported the management of the evaluation consortium and contributed to the process evaluation. She is a mixed-methods researcher specialising in employment, and careers. She has authored several reports for the DWP, including the Evaluation of Fit for Work.

Rebecca Duffy is a Project Support Officer at IES who led proofing and formatting of this report.

Glossary of terms

Term	Definition
Base	The number of observations or cases in a sample. For example, a survey may have a base=2,300 respondents. During analysis the base may become smaller, for example if not all respondents answer a particular question, or when analysing responses from a subset of the full sample.
Baseline data collection	Data from the baseline assessment completed by provider staff who recruited people to the trial.
Benefits	Defined here as the positive or negative impacts for the exchequer or society resulting from the intervention.
Clinical commissioning groups	Clinically-led statutory NHS bodies responsible for the planning and commissioning of healthcare services for their local area.
Controlling for	In statistical modelling with multiple variables and factors, keeping one variable constant in order to examine and test the relationship and effect between other variables of interest in the model.
Dataset	A collection of data or information such as all the responses to a survey or all the recordings from a set of research interviews.
Demographic	A particular section of the population. Also refers to characteristics of an individual of interest for research, such as age, gender, and ethnicity.
Descriptive analysis	Producing statistics that summarise and describe features of a dataset such as the mean, range and distribution of values for variables.
Discount rate	The rate at which future consumption is discounted to reflect society’s preference for present consumption.
EuroQol-5D-5L (EQ5D5L)	Descriptive system for health-related quality of life states in adults, consisting of five dimensions (mobility, self-care, usual activities, pain & discomfort, anxiety & depression), each of which has five severity levels described by statements appropriate to that dimension.
Employment specialists	Staff employed by the trials to undertake randomisation appointments, provide IPS support to the treatment group, and undertake employer engagement.
Final survey	The survey completed by participants 12 months after randomisation.
4-month survey	The survey completed by trial recruits 4 months after starting the trial.
Intervention	The work and health support provided in Sheffield City Region and the West Midlands Combined Authority as part of the trial.
In employment/working	Those in employment full-time, part-time, or less than 16 hours a week; those who are self-employed.
In paid work	Those in employment full-time, part-time, or less than 16 hours a week, not those who are self-employed.
Individual placement and support (IPS)	IPS is a voluntary employment programme that is well evidenced for supporting people with severe and enduring mental health needs in secondary care settings to find paid employment.
Longitudinal surveys	Repeated surveys that study the same people over time.
Multi-morbidity	The occurrence of multiple chronic conditions within the same individual with no single condition holds priority over any of the co-occurring conditions. This term has been selected as the evaluation consortium does not hold information about the main condition affecting individuals.
Participants	Trial recruits allocated to treatment, who went on to receive support, as indicated by having 2+ meetings with an employment specialist. This is used in the 4-month impact analysis chapter (Chapter 6) to differentiate those who experienced limited support beyond randomisation, as in the impact evaluation intention to treat is the basis for analysis. Other terms are used to describe people taking part in the trial (recruits) and people taking part in the surveys (respondents) – see below.
Prevalence	The extent to which something occurs in a population or group, often expressed as a percentage.
Provider staff	Those working in provider organisations including employment specialists delivering IPS support, as well as managers and administrators.
p-value	Used as a measure of statistical significance. Low p-values indicate results are very unlikely to have occurred by random chance. p<0.05 is a commonly cited value, indicating a less than 5% chance that results obtained were by chance. Research findings can be accepted with greater confidence when even lower p-values are cited, for example p<0.01 or p<0.001.
Randomised controlled trial	A study to test the efficacy of a new intervention, in which participants are randomly assigned to two groups: the intervention group receives the treatment, while the control group receives either nothing or the standard current treatment.
Recruits	People who agreed to take part in the trials and who were randomised to either the treatment or control group.
Respondents	Trial recruits from the treatment or control group who were invited to take part in the evaluation and took part in the surveys. As such the descriptive analysis of the survey identifies treatment group respondents and control group respondents.
Short Warwick-Edinburgh mental wellbeing scale (SWEMWBS)	The SWEMWBS is a short version of the Warwick–Edinburgh mental wellbeing Scale (WEMWBS). The WEMWBS was developed to enable the monitoring of mental wellbeing in the general population and the evaluation of projects, programmes and policies which aim to improve mental wellbeing.
Site	The trials were delivered in two combined authorities, which are termed sites.
Social time preference rate (STPR)	See definition of the discount rate.
Statistical significance	Statistical significance indicates that the result or difference obtained following analysis is unlikely to be obtained by chance (to a specified degree of confidence) and that the finding can be accepted as valid. A study’s defined significance level is the probability of the study rejecting the null hypothesis (that there is no relationship between two variables), demonstrated by the p-value of the result.
Survey	A research instrument used to collect data by asking scripted questions or using lists or other items to prompt responses. Can be conducted in person face-to-face, by telephone, or by postal or web-based questionnaire.
Trial arm	This is used to denote the allocation of individuals to either the treatment or control group, with these groups known as the trial arms.
Trial group(s)	Three trial groups are referred to in the report: two out-of-work (OOW) groups (one in each combined authority), and an in-work (IW) group in Sheffield City Region (SCR). These groups are pooled as all OOW and all SCR in different elements of the analysis.
Trial-specific costs	The costs of running a trial which were additional to the costs of delivering the IPS services to those recruited to the treatment group.
Variable	A variable is defined as any individual or thing that can be measured.
Weighting	During analysis of survey data, adjusting for over or under-representation of particular groups, to ensure that the results are representative of the wider population.

1. The 12-month economic evaluation

This report presents the findings of a cost-benefit analysis of the voluntary IPS service, introduced as part of the HLT in Sheffield City Region and West Midlands Combined Authority.

1.1 Introduction

This report draws on findings from the wider evaluation of the HLT, including the process (implementation) and impact studies, to calculate the delivery costs of the trials and IPS services and to attach a monetary value to the estimated benefits of the intervention (IPS) to the exchequer and to society.

The analysis assesses the ratio of benefits to costs over a period of 3 years following the point when participants were randomised to the IPS service. The impact study considered whether IPS had a discernible impact on outcomes for participants 12 months after randomisation. It is probable that any impacts would be sustained beyond this point, but as there is increasing uncertainty over time in how effects might evolve, the economic evaluation focuses on estimating the BCR over a 3-year period. Seeking to estimate the ratio of benefits to costs beyond this point would be likely to increase the BCR, but confidence in the accuracy of the findings would be reduced.

The economic evaluation also explores the sensitivity of the findings to varying the assumptions underlying the analysis and notes any potential costs or benefits which cannot be assessed or valued in monetary terms. As well as sitting alongside the other evaluation reports, this report is accompanied by a spreadsheet which shows the calculations used in the analysis.

The following section provides a brief description of the approach used to assess the costs and benefits of the IPS services. It also sets out the questions that the report seeks to answer. The concluding section provides an overview of the content and coverage of the report.

1.2 Overview of cost-benefit analysis

The economic evaluation seeks to assess both the financial and the economic cases for the IPS services, as implemented through the HLT. A financial appraisal considers the costs and benefits of a programme to the exchequer, whilst an economic appraisal takes into account wider costs and benefits to society (HM Treasury and Welsh Government 2018). This report follows the approach to financial and economic appraisal set out in the HM Treasury Green Book (HM Treasury 2020) and seeks to answer the following questions:

• How much did it cost to deliver the IPS services per recruit in the treatment group? Did the costs vary between trial groups?

• What were the monetary benefits of the IPS services and were there likely to be other benefits that could not be valued in monetary terms?

• What was the ratio of benefits to costs for the financial and economic cases? How is the BCR affected by varying the assumptions underlying the analysis?

The impact evaluation estimated the impact of receiving the IPS services on a number of trial groups. In WMCA the trial was open only to those who were OOW at the time of randomisation, whereas SCR additionally allowed those who were IW to take part. It was therefore possible to estimate the impact of IPS on the SCR IW group, as well as the OOW groups in both SCR and WMCA, both separately and for those initially OOW across both trial sites combined. Whilst impact estimates were also produced for the pooled SCR trial groups – that is, those from the IW and OOW groups – the combined SCR trial group is not considered in the economic evaluation due to the limited generalisability of estimating costs and benefits for trial recruits in very different circumstances at the time of randomisation. In assessing the financial and economic cases, the analysis distinguishes between each of the trial groups considered in the impact evaluation.

1.3 Structure of this report

The following chapter considers the benefits expected to result from the IPS services, both for the exchequer and society. Prior to conducting the analysis, it was uncertain whether some of the effects from receipt of IPS were likely to be positive or negative over a three-year period. Whether the monetary valuation of impacts was positive or negative could also differ as the assumptions underlying the cost-benefit analysis were varied. Chapter 2 therefore considers any impacts arising from receipt of IPS irrespective of whether they are positive or negative for the exchequer or society. Having set out initial expectations regarding the potential effects of the IPS services,

Chapter 2 goes on to summarise the findings of the impact evaluation and to explain the assumptions underlying the calculations of monetary benefits and the steps taken to test the sensitivity of the findings to varying these assumptions. It describes the sources used to attach a monetary value to the impact estimates and concludes by showing the calculations of financial and economic benefits from the IPS services.

Chapter 3 sets out the costs of delivering the IPS services to the treatment group, excluding any trial costs arising from engagement with the control group. It begins by describing the sources of information on delivery costs and then reports on each of the components of costs – namely those incurred at site level and those related to delivering IPS services at national level. The final section provides an estimate of the total costs of delivery and the costs per recruit to the treatment group (1,260 SCR IW, 1,799 SCR OOW and 1,837 WMCA).

Chapter 4 combines the estimated monetary benefits of the IPS services with the information on the costs of delivery to calculate the BCR for both the financial case and the economic case. To place these findings in context, it describes any elements of costs and benefits which were likely to be omitted due to difficulties in estimation or in attaching a monetary value to known impacts. The chapter concludes by summarising the results of sensitivity testing and explaining how varying the assumptions underlying the analysis affects the results.

The final chapter summarises the main findings of the analysis and describes its limitations, to highlight areas of uncertainty. This includes a consideration of how any potential impacts which cannot be valued in monetary terms might affect the overall conclusions drawn on the cost effectiveness of the IPS services.

2. Valuing the impact of IPS

This chapter provides estimates of the monetary benefits of IPS services across the trial groups, namely the SCR IW and SCR OOW groups, the WMCA OOW group and the pooled SCR-WMCA OOW group.

2.1 Introduction

Benefits from the IPS services are estimated both in terms of their monetary value to the exchequer and to society. Monetary benefits are reported in terms of their estimated total value as well as per recruit to the treatment group. As the cost-benefit analysis is based on a series of assumptions, the results are presented for 3 scenarios:

• The reference scenario (the main focus in the cost benefit analysis (CBA)) is based on assumptions commonly used in other cost-benefit analyses.

• The optimistic scenario uses assumptions at the upper end of reasonable expectations.

• The pessimistic scenario is based on assumptions which are towards the lower end of expectations.

The detailed assumptions made under each of these scenarios can be found in Table A.1 in Appendix A.

Section 2.2 lists the expected benefits of receiving IPS, whilst section 2.3 summarises the impact estimates found in the impact study (see the 12-month impact report). It also considers how these impacts might be sustained over time, in part informed by further analysis conducted as part of the economic evaluation.

Section 2.4 lists the assumptions underpinning the modelling whilst section 2.5 discusses the sources and methodology used to attach values to the impact estimates. The chapter concludes by showing the results of the modelling of the financial and economic benefits of IPS.

2.2 Expected effects

The definition of benefits used in this chapter encompasses any effect from receiving the IPS service on the exchequer or society as a whole. When valued in monetary terms these effects can be positive; for example, an increase in earnings for members of the treatment group; or negative, such as an increase in pollution from additional work related travel. A benefit can be positive from one perspective, but negative from another. For instance, a reduction in the amount of Universal Credit (UC) paid to a trial recruit would be negative from the perspective of the recruit, but positive from the perspective of the exchequer.

Table 2.1 outlines the anticipated effects of IPS receipt for the financial or economic cases and whether they were expected to be positive or negative. Benefits for those who were in-work (IW) or out-of-work (OOW) at the time of randomisation are considered separately. Appendix B provides a detailed description of each of the potential benefits. As randomisation of the IPS service commenced in May 2018, the base period used in calculating benefits is the 2018 to 2019 financial year (running from 6 April 2018 to 5 April 2019). The approach to monetising the estimated impact of IPS is explained in section 2.5.3.

Table 2.1 Potential financial and economic benefits for those in, or out-of-work

Potential benefit from IPS receipt	Financial (in-work)	Financial (out-of-work)	Economic (in-work)	Economic (out-of-work)
Change in productivity			positive	positive
Change in UC and legacy benefit payments	positive	positive
Change in other benefit payments and tax credits	positive	positive
Change in DWP operational costs	positive	positive	positive	positive
Change in income tax revenue	positive	positive
Change in employee National Insurance contributions (NICs)	positive	positive
Change in employer NICs	positive	positive
Change in revenue from indirect taxes	positive	positive
Change in healthcare costs	positive	positive	positive	positive
Change in travel costs			negative	negative
Change in childcare costs			negative	negative
Change in health-related quality of life			positive	positive

2.3 Observed impacts

The impact report provides the estimated impact of the IPS service. This section focuses on findings relevant to the economic evaluation, with all results drawn from the 12-month impact analysis, unless stated otherwise. The dataset used to produce the impact estimates included linked DWP and HMRC administrative records as well as data from survey interviews conducted with recruits around 12 months after randomisation.

As the cost-benefit analysis considers benefits over a period of 3 years after randomisation, it is necessary to consider how the impact estimates from the first 12 months following randomisation might evolve over a longer period. The approach to this is explained below.

The analysis focuses on benefits over a period of up to 3 years after randomisation due to the increasing uncertainty surrounding the magnitude of benefits over time. However, it is possible that benefits will continue to accrue after this point. Providing the total monetary value of benefits from the trials is positive, the estimates of the BCRs are likely to be an understatement of the true figure, given that there are no ongoing costs from IPS service delivery beyond 3 years after randomisation.

2.3.1 Change in employment

Many of the expected benefits from receipt of IPS are related to whether participation had an impact on employment. The impact analysis found that there were positive and statistically significant effects on the measure of sustained employment in paid work (whether an individual was in paid work for 13 weeks or more in the 12 months following randomisation) for the SCR IW and WMCA trial groups. For the SCR IW group there was a 3.2 percentage point (ppt) impact on the employment outcome which was statistically significant at the 90% confidence level. For the WMCA trial group there was a 3.8 ppt impact on sustained employment which was statistically significant at the 99% level of confidence. There was no discernible impact on sustained employment for the SCR OOW trial group, or for the pooled OOW groups. There were also no statistically significant impacts on earnings or benefit receipt for any of the trial groups.

Whilst the main aim in the impact evaluation was to estimate impacts for a period of 12 months following randomisation, data supplied by HMRC covered 22 months following randomisation for all trial recruits. These data were used in the economic evaluation to assess how impacts evolved for a further 10 months. This analysis focused on impacts on the proportion of time employed within the first 12 months following randomisation, and then over the full 22 months observed. For the SCR IW group, time spent in employment in the 12 months following randomisation increased by 1.5 weeks as a result of the IPS service. Over the 22 months following randomisation, time spent in employment rose by 2.3 weeks for the SCR IW group due to IPS. For the WMCA trial group, time spent in employment in the year following randomisation was 1.1 weeks higher as a result of IPS. Over a 22-month period, time spent in employment for the WMCA trial group was 2.6 weeks higher due to IPS.

To estimate the impact of IPS on employment over the period from 12 to 24 months after randomisation, it was assumed that the change in impact seen between 12 and 22 months continued to follow the same trend up to the 24-month point. The reference scenario assumed that between 24 and 36 months after randomisation the impact of IPS on employment was the same as that estimated between 12 and 24 months after randomisation. In WMCA the optimistic scenario assumed that the upward trend in impact seen in the second year following randomisation continued in the third year, whilst the pessimistic scenario assumed that in the third year following randomisation the impact of the IPS service was the same as the average impact across the first 2 years following randomisation. As the employment impact followed a downward trend between 12 and 24 months following randomisation, the definitions of the optimistic and pessimistic scenarios were switched. Thus, the pessimistic scenario assumed that the downward trend in impact seen between 12 and 24 months continued in months 24 to 36, whilst the optimistic scenario assumed that the impact in months 24 to 36 was the average seen in the first 2 years following randomisation.

2.3.2 Change in health

The impact analysis, using the 12-month survey data, found that receipt of IPS had a positive and statistically significant impact on health for the SCR IW, SCR OOW and all OOW trial groups. Raw scores on the primary health measure (EQ5D5L), which captures five different dimensions of health, were converted to a recognised measure of utility derived from each health state. For both the SCR IW and SCR OOW groups, there was a 0.03 unit increase. This finding was statistically significant at the 90% level of confidence for both of these trial groups. In the combined OOW group, there was a 0.02 unit increase which was statistically significant at the 95% level, although there was no clear impact on health in WMCA.

As the health outcomes were drawn from the final survey of the trial, it was not possible to estimate health impacts over a longer period to inform the definitions of the alternative scenarios used to assess the sensitivity of the findings to varying the underlying assumptions. However, the scenarios are guided by a substantial increase in impacts seen between the 4-month interim impact evaluation and the 12-month final impact evaluation. The impact of the IPS service on the primary health outcome was less than 0.01 units for the SCR IW trial group and the all OOW group 4 months after randomisation. For the SCR IW trial group this impact was statistically significant at the 99% level of confidence, whilst it was statistically significant at the 90% level for the all OOW trial group. No statistically significant impact on the primary health outcome was apparent for the SCR OOW or WMCA OOW groups 4 months after randomisation.

The reference scenario for health impacts assumes that the year 1 impacts on the primary health measure were sustained into years 2 and 3. This assumption means that the benefits to health do not wear off after the individual stops receiving IPS, but there is also no additional benefit from the further application of health condition management techniques learnt through the programme in later years. The optimistic scenario assumes that there is a 50% rise in health impacts year to year, whilst the pessimistic scenario assumes that health impacts fall by 50% year to year. This impact is monetised using quality-adjusted life years (QALYs). This measure of the burden of ill health is calculated by multiplying the change in quality of life due to an intervention (in this case the IPS services in the HLT) by the duration of the impact. The details of how this is done are outlined further in Appendix B.

The impact report included estimates of the impact of the IPS service on wellbeing. This was considered as one of the primary outcome measures. The IPS service had a positive impact on wellbeing for the IW and OOW groups in SCR, although this was only at the 90% level of confidence for the SCR OOW trial group. The impact on wellbeing in WMCA was indeterminant, although when the OOW groups from SCR and WMCA were pooled, a positive impact on wellbeing across the 2 groups was apparent. Monetising the impact of IPS on multiple health measures would result in the double counting of benefits, so instead the economic evaluation focuses on monetising impacts on the primary health outcome for the trial rather than also covering the impacts on the secondary outcomes of disability and musculoskeletal health. For the same reason the analysis does not seek to monetise impacts on wellbeing and mental health for the SCR IW, SCR OOW and the combined OOW trial groups. Although EQ5D5L focuses on functional health rather than mental health and wellbeing, one of the five dimensions does relate to mental health (that is, the anxiety/depression dimension). As a result, the economic evaluation does not seek to monetise wellbeing impacts in combination with impacts on general health to avoid double counting.

2.4 Assumptions underlying the calculations

2.4.1 Cohort structure

Trial recruits were randomised between 8 May 2018 and 31 October 2019. To apportion benefits by financial year, recruits randomised in the 2018 to 2019 financial year (6 April 2018 to 5 April 2019) were treated as the first cohort, and those randomised in the 2019 to 2020 financial year (6 April 2019 to 5 April 2020) were regarded as the second cohort.

Effectively the analysis assumes that every individual in the first cohort was randomised on 6 April 2018, and every individual in the second cohort was randomised on 6 April 2019. This is necessary to accrue benefits to a 3-year period following randomisation, given that recruits were randomised over a period of 2 financial years. Also, the analysis assumes that 12-month impacts occur within the same financial year as randomisation. This is because the economic evaluation is concerned with how the IPS services affected the treatment group and the benefits that accrue as a result, rather than the precise timing of the trial. Financial years are used to ease the process of monetising benefits, rather than to provide an accurate indication of how benefits accrued from a trial carried out at a particular point in time.

2.4.2 Discount rates

For the potential employment-related impacts (change in employment status, change in earnings and change in benefits received), the Green Book’s STPR of 3.5% is applied in the reference scenario (HM Treasury 2020). This adjusts for society’s preference for present consumption, rather than consumption deferred to a future date and is also known as the discount rate.

For the increase in health-related quality of life (HRQOL), the yearly discount rate of 1.5% as recommended for QALY effects according to the Green Book is used. The method used to monetise these benefits, detailed in Annex A1 of the Green Book (HM Treasury 2020), is applied in the reference scenario.

In the optimistic scenario, the discount rate for the employment-related benefits is reduced to 3%, and in the pessimistic scenario it is raised to 4%. In the optimistic scenario, the discount rate applied to the HRQOL impacts is unchanged, but in the pessimistic scenario it is increased to 3.5%. The National Institute for Health and Care Excellence (2012) recommend varying the discount rate applied in QALY calculations to 3.5% in sensitivity analysis.^{[footnote 1]}

2.4.3 Inflation rates

Prices used in the monetisation of benefits are forecast forward using the April to April consumer price inflation (CPI) rate provided by the Office for National Statistics (2022) up to 2020 to 2021. The forecast CPI rate for 2021 to 2022 is taken from the Office for Budget Responsibility’s Economic and Fiscal Outlook for October 2021 (Office for Budget Responsibility, 2021). The financial year rate is constructed by aggregating the rates for 2021Q2, 2021Q3, 2021Q4 and 2022Q1. The analysis also assumes that the CPI rate is representative of the increase in the price for the benefit under consideration.

The forecast rate is uncertain and so is subjected to sensitivity analysis. In the optimistic scenario it is increased to 3.8%, and in the pessimistic scenario it is reduced to 2.8%. The realised inflation rates up to 2020 to 2021 and the forecast 2021 to 2022 inflation rate used to convert nominal values into real ones can be found in the spreadsheet which accompanies this report.

2.4.5 Distributional effects

Some economic evaluations incorporate distributional effects into their benefit calculations on the basis that the utility gain from any additional income depends on the individual’s initial income level. When monetising the benefits of a programme such as IPS, benefits that accrue to individuals with a lower income level are likely to result in greater welfare gains than those that accrue to individuals with a higher income level. However, as the IPS services had no discernible impact on earnings, no distributional effects were likely for the HLTs.

2.4.6 Social Cost of Exchequer Finance

Taxation can affect the demand and supply of labour for the following reasons:

• income tax reduces the net pay an employee receives from work and so they may adjust the hours they choose to work as a result

• employer National Insurance Contributions (NICs) increase the net cost of employing someone and so they may choose to employ fewer workers than if there were no employer NICs

• indirect taxes increase the cost of consuming goods and services to which they are applied, and so individuals and firms may choose to consume less of these products

As public programmes such as the IPS service in the trials are funded through additional tax revenues, this has an impact on economic efficiency. This is known as the Social Cost of Exchequer Finance (SOCEF). The SOCEF is set to zero in the reference and pessimistic scenarios and 20% in the optimistic scenario. This is because the net financial benefits (total benefits minus total costs) are negative for each trial, meaning that applying a SOCEF of 20% is a positive benefit.

2.4.7 Impact estimates

Several of the impact estimates used in this economic evaluation are based on analysis of the trial’s survey data. These surveys took place at randomisation and then at around 4 and 12 months following randomisation. All recruits were invited to take part at each point, unless they had withdrawn from primary research.

As is common with longitudinal surveys, there was substantial attrition. All recruits completed the baseline survey, but the survey carried out 12 months after randomisation had response rates of between 41.0% and 50.1% for the treatment group and between 36.1% and 42.5% for the control group across the trial groups (Dorsett et al. 2022, 12). As the observed benefits from the programme are monetised based on the total number of recruits in the treatment group and not just those who responded to the final survey, it is assumed that the impact estimates derived from the responses to the final survey (and the interim survey when used, as mentioned in section 2.3.2) are representative of all recruits to the treatment group. This may not be the case if there is systematic attrition between the trial arms. In the impact evaluation, estimates based on responses to the final survey were weighted in an attempt to address any biasing effect. This appeared to have been effective for the SCR OOW group and in WMCA, but imbalances remained for the SCR IW group even after weighting (see 12-month impact report for more information).

As with all impact evaluations, the impact estimates are subject to uncertainty. For this reason they are subjected to sensitivity analysis through the application of Monte Carlo simulations reported in Chapter 4.

2.5 Monetising impacts

2.5.1 Attaching monetary values to the estimated impacts

Table 2.2 displays the per unit valuations of the outcomes that enter our calculations of the financial and economic benefits of the Health-led Trials, as well as the year of the valuation. As mentioned earlier, the values are adjusted to take account of price inflation since the valuation was made. The table also reports the source of each of the values. How these figures were derived is described in the text below the table.

Table 2.2 Per unit valuations for financial and economic analyses

Outcome	Unit valuation - Financial	Unit valuation - Economic	Price year	Difference between financial and economic valuations
Reduction in healthcare costs for an individual moving from unemployment to full-time employment	£508 per non-ESA individual per year £1016 per ESA individual per year	£508 per non-ESA individual per year £1016 per ESA individual per year	2008 to 2009	Any additional economic effects are not considered
Increase in childcare costs for an individual moving from unemployment to full-time employment	-	£744 per lone parent per year £694 per couple with children per year	2008 to 2009	Any potential benefit accrued by the exchequer is not considered
Increase in travel costs for an individual moving from unemployment to full-time employment	-	£433 per year	2008 to 2009	Any potential benefit accrued by the exchequer is not considered
Improvement in health (quality measured by EQ5D5L)	-	£60,000 per QALY	Fixed	Any potential benefit accrued by the exchequer is not considered

Base: 2018 to 2019

Source: Fujiwara (2010); HM Treasury (2020).

The outcome measure used in the economic evaluation to capture employment effects was the number of months employed in the 12 months following randomisation (a monthly figure), whereas the figures used to monetise the change in employment status (relating to healthcare costs, childcare costs and travel costs) are annual. These annual figures are transformed to monthly figures by dividing by 12, hence we are assuming that these figures can be apportioned equally over a period of 12 months. However, this may not be the case. For example, there may be some fixed component to travel costs relating to annual parking permits, annual rail fares and so on. As such, these values will probably be understated when considered on a monthly basis if the impact is small. As some of these elements are positive (reduced healthcare costs) and some are negative (increased childcare costs, increased social travel costs), it is not clear which way the total benefit figures will be skewed by this.

The figures used to monetise the increase in travel costs and childcare costs and reduction in healthcare costs are national averages. Assuming that this is representative of the true costs of travel and childcare in SCR and WMCA may be inaccurate. For instance, individuals in urban areas such as SCR and WMCA are likely to have shorter commutes with better and more frequent public transport links (Fujiwara 2010), hence the true travel costs may be lower than the figures shown in Table 2.2.^{[footnote 2]}

Additionally, as the IPS services were targeted at individuals with pre-existing health conditions, figures based on national averages may be particularly unrepresentative, especially for the reduction in healthcare costs. The value used in the monetisation of benefits was derived by combining 2 figures – a higher value for those claiming ESA and a lower value for those not claiming ESA - from Fujiwara (2010). Individuals are eligible for ESA if they have a disability or health condition that affects how much they can work, so those eligible for ESA are likely to have higher healthcare service utilisation than those not eligible to claim ESA. Moving this group into employment, or to increased hours for those already in work, would be likely to reduce demand for healthcare services and increase benefits accrued by the exchequer and society.

However, recruits who do receive ESA are likely to have a greater prevalence of health conditions than those in the general population who do not receive ESA, given that the trial was targeted at those with health problems. It is therefore possible that the analysis underestimates the reduction in healthcare costs associated with a change in employment status. Additionally, the benefits data available to the economic evaluation only provided information on the proportion of recruits claiming ESA up to 21 months after randomisation for the year 1 cohort, and only 14 months after randomisation for the full cohort. It is therefore necessary to make assumptions about how these proportions evolve after this point up to 3 years post-randomisation.

The figures used to monetise a change in employment status and associated reduction in healthcare costs, increase in childcare costs and increase in travel costs, come from the DWP’s Social Cost Benefit Analysis Framework, again adjusted in line with price inflation since the valuation was made Fujiwara (2010). Weighted values are constructed for the reduction in healthcare costs and increase in childcare costs using the proportion of ESA claimants, and the proportion of lone parents and couples with children in the baseline data.

It is possible to construct monthly proportions of those claiming ESA up to 21 months post-randomisation for the year 1 cohort. This is used to forecast the proportion of ESA claimants up to 36 months after randomisation and the monthly proportions are aggregated to annual figures for 1, 2 and 3 years post-randomisation. The baseline proportions of lone parents and couples with children and annual proportions of ESA claimants over time can be found in Appendix C.

The value attached to increased childcare costs in the monetisation of benefits was derived by combining 2 figures, a higher value for lone parents and a lower value for couples with children. As data on the proportions of lone parents and couples with children were only available at baseline, it is necessary to assume that these do not change over time.

The employment-related figures in the table are calculated by Fujiwara (2010) in the DWP’s Social Cost Benefit Analysis Framework based on an individual moving from unemployment into full-time employment. However, some of the recruits who entered employment as a result of IPS secured part-time jobs. To account for this, the calculations are adjusted by the average number of weeks worked in a month by the treatment group for each of the trial groups considered (reported in Appendix C).

Additionally, the childcare and travel cost figures include the impact of increased earnings due to gaining employment on an individual’s patterns of consumption for these 2 services. As discussed, the impact evaluation found no earnings impact alongside the employment impact in WMCA or for the SCR IW trial group. Therefore the figures from Fujiwara (2010) would overestimate the true increase in costs incurred as a result of this trial in WMCA. This is not the case for the healthcare cost figures because of the strong evidence that income has a relatively small effect on health. The childcare and travel cost figures are discounted to allow for the role played by increased income in their derivation. The reference scenario assumes that increased incomes are responsible for 25% of the increase in costs faced, and then 50% and 10% in the optimistic and pessimistic scenarios respectively. To be conservative in the estimates of benefits, the pessimistic scenario assumes increased incomes are responsible for 25% of the reduction in healthcare costs.

It was not possible to gain access to NHS Digital Data which would have made it possible to monetise the impact of the trials on health by considering difference in the utilisation of healthcare services between the treatment and control groups. This is considered in the model through the effect of a change in employment status on health and thus the utilisation of healthcare services, but is unlikely to fully capture these effects for the treatment group given the higher morbidity and multi-morbidity rates of the trial population compared with the general population.

A QALY-based approach was used to directly monetise the observed change in general health on the primary health measure (EQ5D). The raw scores of respondents to the 12-month survey were converted to a measure of HRQOL using an established approach.^{[footnote 3]} QALYs are calculated by multiplying the change in quality of life by the duration in years of the change. These are monetised in the reference and pessimistic scenarios using the value of £60,000 per QALY outlined in The Green Book (HMT, 2020). For example, an improvement in quality of life from 0.5 to 1.0 for 6 months equals the gain of 0.25 QALYs which has an economic value of £15,000. In the optimistic scenario, the value of £60,000 per QALY will be increased over time using the gross domestic product (GDP) deflator (HM Treasury 2022), the values of which can be found in the spreadsheet which accompanies this report.

The primary health measure used in the impact evaluation captured impacts at the point 12 months after randomisation and not over the whole 12-month period. To reflect the duration component of QALYs it is necessary to consider the evolution of the impact over time. Instead of simply assuming a linear trajectory in impact from no impact at randomisation to the full impact realised 12 months after randomisation, the proportion of the 12-month impact felt after 4 months based on estimates from the implementation, and the 4-month outcomes report, are used to produce estimates of the realised benefit for the first year of the trial.^{[footnote 4]}

2.5.2 Financial benefits

Table 2.3 displays the present value (in 2018 to 2019 prices) of financial benefits by year and the total value of financial benefits across the 2 trial treatment groups where there was some evidence of impact on employment from participation in IPS; namely SCR IW and WMCA. The impact on HRQOL does not constitute a financial benefit, for the reasons set out in Appendix B. Results are shown for the reference, optimistic and pessimistic scenarios. As there was no clear employment impact for the SCR OOW or all OOW trial groups there are no financial benefits from the HLT for these and so they are omitted from the table.

As there were statistically significant employment impacts for the WMCA and SCR IW trial groups, there were estimated financial benefits. To compare the value of the financial benefits, it is most meaningful to look at returns per recruit to the treatment group. These are presented in Table 2.4 which displays yearly and total present value (in 2018 to 2019 prices) of financial benefits per recruit in the SCR IW and WMCA trial groups for the reference, optimistic and pessimistic scenarios. The number of recruits to the treatment group changes yearly based on the cohort structure discussed in section 2.4.1.

Table 2.3 Present value of financial benefits for the SCR IW and WMCA trial groups, total and by year

SCW IW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£10,191	£18,579	£14,790	£7,966	£51,526
Optimistic	£10,191	£18,669	£16,418	£9,875	£55,153
Pessimistic	£6,115	£11,094	£7,042	£2,664

WMCA trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£11,751	£29,188	£33,714	£17,723	£92,376
Optimistic	£11,751	£29,329	£39,036	£23,763	£103,879
Pessimistic	£7,051	£17,428	£18,565	£8,788	£51,832

Source: Economic evaluation spreadsheet.

Table 2.4 Present value of financial benefits for the SCR IW and WMCA trial groups per recruit to the treatment group, total and by year

SCW IW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£18	£15	£12	£12	£41
Optimistic	£18	£15	£13	£15	£44
Pessimistic	£11	£9	£6	£4	£21

WMCA trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£14	£16	£18	£18	£50
Optimistic	£14	£16	£21	£24	£57
Pessimistic	£8	£9	£10	£9	£28

Notes: The present value of benefits each year does not sum to the total present value of benefits per recruit as estimates for each year reflect the composition of the cohort in that particular financial year.^{[footnote 5]}

Source: Economic evaluation spreadsheet

In the reference scenario, the total financial benefits per recruit to the treatment group were estimated to be £41 and £50 for the SCR IW and WMCA trial groups respectively. For the SCR IW trial group the financial benefit per recruit decreased over time reflecting the downward trend in the number of additional weeks employed over time. Conversely, in WMCA the financial benefit per recruit rose over time as the impact on the number of additional months employed increased. In the optimistic scenario, the total financial benefit per recruit was estimated to be £44 and £57 for the SCR IW and WMCA trial groups respectively. In the pessimistic scenario, the total financial benefit per recruit was estimated to be £21 and £28 for the SCR IW and WMCA trial groups respectively. The entirety of the financial benefits in the reference and optimistic scenarios came from the reduction in healthcare costs associated with an increase in employment. However, in the pessimistic scenario substitution effects reduced the financial benefit as the increase in employment by recruits reduces employment opportunities for others.

Substitution effects are a negative benefit equivalent to 20% of the value of the reduction in healthcare costs in the pessimistic scenario (see section 2.4.4). Introducing substitution effects of 20% to the reference scenario reduces the total present value of financial benefits per participant in the SCR IW and WMCA trial groups to £33 and £40 respectively. By contrast, varying the employment-related discount rate and the inflation forecast has a negligible impact.

Switching the reference scenario to use the employment impact trajectory assumptions of the optimistic scenario would result in total financial benefits per recruit to the treatment group of £43 and £56 for the SCR IW and WMCA trial groups respectively, compared to figures of £36 and £47 respectively when using the assumptions of the pessimistic scenario. Finally, assuming that the proportion of healthcare costs that is due to an earnings impact is 25% reduces the figures for the SCR IW and WMCA trial groups to £31 and £38 respectively. Thus, the main contributors to the variation in financial benefits for the SCR IW and WMCA trial groups between the 3 scenarios are the proportion of healthcare costs due to an assumed earnings impact, substitution effects, and assumptions about the trajectory of the employment impact.

2.5.3 Economic benefits

Table 2.5 displays the present value (in 2018 to 2019 prices) of economic benefits by year and the total value of economic benefits across the SCR IW, SCR OOW, WMCA and all OOW trial groups for the reference, optimistic and pessimistic scenarios. Despite the SCR OOW and all OOW groups showing no financial benefits (due to a lack of monetisable employment impact), there were positive economic benefits, due to the impacts on health for the SCR IW, SCR OOW and all OOW trial groups, resulting in substantial HRQOL gains. The all OOW trial group had the greatest total economic benefits, but this is partly because it was the largest group of trial recruits.

SCW IW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£823,572	£1,950,850	£2,139,378	£1,137,535	£6,051,336
Optimistic	£1,159,373	£2,628,809	£4,348,079	£2,757,357	£10,893,618
Pessimistic	£819,852	£1,416,739	£788,477	£266,916	£3,291,985

SCW OOW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£935,113	£2,606,146	£3,269,616	£1,733,305	£8,544,180
Optimistic	£1,469,169	£3,652,999	£6,660,124	£4,201,924	£15,984,216
Pessimistic	£935,113	£1,804,114	£1,209,111	£408,688	£4,357,026

WMCA trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£2,111	£4,860	£5,066	£2,545	£14,582
Optimistic	£764,652	£439,576	£175,128	£5,443	£1,384,798
Pessimistic	-£2,204	-£5,814	-£6,660	-£3,253	-£17,931

All OOW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£1,635,935	£4,196,901	£4,966,376	£2,614,704	£13,413,916
Optimistic	£2,931,669	£6,129,999	£10,239,438	£6,338,635	£25,639,740
Pessimistic	£1,635,935	£2,964,910	£1,832,160	£616,509	£7,049,515

Source: Economic evaluation spreadsheet.

Table 2.6 displays the yearly and total present value (in 2018 to 2019 prices) of economic benefits per recruit to the treatment group receiving IPS across the trial groups for the reference, optimistic and pessimistic scenarios.

Table 2.6 Present value of economic benefits per recruit to the treatment group, total and by year

SCW IW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£1,420	£1,548	£1,698	£1,673	£4,803
Optimistic	£1,999	£2,086	£3,451	£4,055	£8,646
Pessimistic	£1,414	£1,124	£626	£393	£2,613

SCW OOW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£1,125	£1,449	£1,817	£1,791	£4,749
Optimistic	£1,768	£2,031	£3,702	£4,341	£8,885
Pessimistic	£1,125	£1,003	£672	£422	£2,422

WMCA trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£2	£3	£3	£3	£8
Optimistic	£887	£239	£95	£6	£754
Pessimistic	-£3	-£3	-£3	-£3	-£10

All OOW trial group and scenario	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reference	£966	£1,154	£1,366	£1,346	£3,689
Optimistic	£1,732	£1,686	£2,816	£3,262	£7,052
Pessimistic	£966	£815	£504	£317	£1,939

Notes: Present value of benefits each year does not sum to total present value of benefits per recruit as estimates for each year reflect the composition of the cohort in that particular financial year.

Source: Economic evaluation spreadsheet.

The 2 SCR trial groups (IW and OOW) produced the greatest economic benefit at the level of the recruit. These are remarkably similar in the reference scenario in particular. The total economic benefits per recruit in the treatment group of the SCR IW and SCR OOW trial groups were estimated to be £4,803 and £4,749 respectively in the reference scenario, compared with estimates of £8,646 and £8,885 respectively in the optimistic scenario, and £2,613 and £2,422 respectively in the pessimistic scenario. The all OOW trial group also produces substantial economic benefits per recruit of £3,689, £7,052 and £1,939 in the reference, optimistic and pessimistic scenarios respectively. However, the WMCA trial group produces significantly smaller returns than the others on a per recruit basis. This is largely due to the lack of impact from the IPS service in WMCA on health, resulting in a lack of benefits derived from a change in HRQOL. The estimates of the per recruit total economic benefits for WMCA are £8 in the reference scenario, £754 in the optimistic scenario, and £10 in the pessimistic scenario.

For the SCR IW, SCR OOW and all OOW trial groups, the 2020 to 2021 financial year produced the highest economic benefit per recruit. However, in the optimistic scenario, 2021 to 2022 has the highest economic benefit per recruit, with the combination of the GDP deflator being applied to the value of a QALY and the assumption of an increasing health impact over time being largely responsible for this forward shift. In the pessimistic scenario the highest economic benefit occurs in 2018 to 2019, with the assumption of a decreasing health impact over time being largely responsible for this backward shift.

Figures 2.1 to 2.4 display the contribution of each source of economic benefits to the total and annual figures under the reference scenarios for the SCR IW, SCR OOW, WMCA and all OOW trial groups respectively. The improvement in HRQOL is the main driver of the differences in economic benefits between interventions. This was also the case under the optimistic and pessimistic scenarios (shown in Appendix D).

Figure 2.1 Total and annual economic benefits from each source for the reference scenario, SCR IW

Economic case	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reduced healthcare costs	£10,191	£18,579	£14,790	£7,966	£51,526
Increased childcare costs	-£2,682	-£4,886	-£3,886	-£2,093	-£13,547
Increased travel costs	-£6,238	-£11,367	-£9,041	-£4,870	-£31,516
Improved health-related quality of life	£822,301	£1,948,524	£2,137,516	£1,136,532	£6,044,873

Source: Economic evaluation spreadsheet

Figure 2.2 Total and annual economic benefits from each source for the reference scenario, SCR OOW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Improved health-related quality of life	£935,113	£2,606,146	£3,269,616	£1,733,305	£8,544,180

Source: Economic evaluation spreadsheet

Figure 2.3 Total and annual economic benefits from each source for the reference scenario, WMCA

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Increased childcare costs	-£2,721	-£6,866	-£8,085	-£4,284	-£21,956
Increased travel costs	-£6,920	-£17,461	-£20,562	-£10,894	-£55,838
Reduced healthcare costs	£11,751	£29,188	£33,714	£17,723	£92,376

Source: Economic evaluation spreadsheet

Figure 2.4 Total and annual economic benefits from each source by scenario, all OOW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Improved health-related quality of life	£1,635,935	£4,196,901	£4,966,376	£2,614,704	£13,413,916

Source: Economic evaluation spreadsheet

The change in HRQOL accounts for 100% of the total economic benefits for the SCR OOW and all OOW trial groups in the reference and pessimistic scenarios, and 94.7% and 91.4% in the optimistic scenarios respectively. For the SCR IW trial group, whilst there was an employment impact and thus employment-related benefits, these were dwarfed by the change in HRQOL. This accounted for 99.9%, 95.1% and 100.3% of the total economic benefits in the reference, optimistic and pessimistic scenarios respectively (the latter being greater than 100% as the total employment-related benefits are net negative in this scenario).

In WMCA where there was no impact from the IPS service on health and hence no change in HRQOL, the economic benefits are negligible except in the optimistic scenario where the Social Cost of Exchequer Finance drives the results. The reduction in healthcare costs is largely offset by the increase in travel and childcare costs resulting from the employment impact observed in WMCA, as is also the case for the SCR IW trial group.

These findings suggest that for the provision of IPS LITE to generate economic benefits to society, achieving health-related outcomes is necessary. In addition to their small magnitude on the aggregate level, the employment-related benefits are not substantial even when considering that they may be concentrated among certain groups. For instance, the calculations show that the increase in childcare costs in WMCA for the reference scenario on a per family basis (lone parents and couples with children) is £51, reaching a maximum of £19 on an annual basis (2020 to 2021 and 2021 to 2022).

3. Costs

This chapter provides estimates of the financial costs of delivering the IPS services to the treatment group in the 2 trial sites. This involves estimating the costs that result from the IPS delivery net of the costs that would have been incurred if the trial had not taken place.

3.1 Introduction

The chapter focuses on the costs of delivering the intervention to the treatment group and attempts to exclude any costs of designing, delivering and evaluating the trial. Negative impacts that produce a cost to the exchequer or to society have been considered in Chapter 2. It is necessary to identify the costs because the economic evaluation is concerned with estimating the ratio of benefits to costs from the IPS services themselves, rather than the trial. Trial costs include the costs of designing and managing the trials and their evaluation, including the data architecture and information governance required specifically for the purposes of evaluation, rather than only the costs of delivering the IPS services to recruits in the treatment group. All costs are reported as total costs and costs per recruit receiving IPS.

The chapter starts by discussing the sources of costs. Section 3.3 presents information on site costs and the costs sustained by the Work and Health Unit (WHU). It then moves on to show the total costs for each trial site, including a split between the IW and OOW groups in SCR.

Sources of data on programme costs

Data on the costs of delivering the IPS services were gathered from a range of sources. The main source was the WHU, which collected information on costs for SCR and WMCA separately. The synthesis report provides more information on the management arrangements in each of the trial sites.

Information on expenditure was collected as sites claimed back their expenses from the DWP. In SCR expenditure was split into categories including ‘Project Delivery’, ‘Programme Management and Admin’ (split into internal and external costs), and ‘Marketing, Stakeholder Participation and Co-design’. The remaining 3 categories were ‘Consultancy and Temporary Support’, ‘Business Support’ and ‘Contingency’. A more detailed description for each category is found in column 2 of Table 3.1.

In WMCA there was less detailed documentation on the coverage of the cost categories. However, it was possible to distinguish between staffing, Clinical Commissioning Groups (CCGs) support costs, third party costs and IPS provider costs. In SCR, one IPS provider delivered across the area, whereas in WMCA 3 prime providers covered delivery in 3 geographic areas. In WMCA, the provider costs are grouped together to avoid the disclosure of commercially sensitive information. Third party costs were assumed to include IT and business support and any marketing contracted out externally, as well as other items. Differences between local authorities and CCGs in reclaiming VAT meant that some providers in WMCA paid VAT. This was removed from cost calculations in order to focus on real resource costs.

In addition to information collated by WHU, providers were asked to supply information on the number of staff engaged in different types of activities across the period that the trial operated from its design to the end of the support period, 12 months following the final month of recruitment. Hence information covers the period between November 2017 and October 2020. Each of the providers supplied data on their staffing levels within 6-month blocks during this period, as well as data on the proportion of staff time spent on various activities, some related to the delivery of the IPS services and others related to operating as part of a randomised controlled trial.

The economic evaluation also collected in-depth information on staff activities throughout the trial period through a survey of trial staff working at both sites in October 2020. The survey was completed by 15 employment specialists, vocational specialists, and team leaders across all providers in WMCA; and 17 employment specialists, work and enterprise coaches and managers in SCR. Respondents were asked a range of questions about their role in the delivery of the trial. These included questions relating to their contracted and actual working hours throughout their time working on the trial, time spent generating referrals, time spent on appointments and the frequency of appointments, time on employer engagement each week, average caseloads and travel time to face-to-face appointments.

The WHU were also asked to provide a list of their own staff who worked on the trial, including start and end dates, grade and their role in the trial. This information was used to calculate WHU staffing costs, including pension and NI contributions, based on publicly available sources, as well as information supplied by WHU on salary scales.

Information from all of these sources was compiled to estimate intervention-specific costs. In the analysis below these are broken down into set-up and ongoing costs for each site as well as WHU costs and then added together to give an estimate of total costs for the trials and IPS services. The results presented in section 3.3 are reported as total costs as well as costs per recruit in the treatment group.

Components of costs

Funding for the HLT was governed by agreements in the form of Grants and Memoranda of Understanding via the WHU Innovation Fund. This itself was funded by the DWP, the Department for Health and Social Care (DHSE) and NHS-E. Initial ‘seed’ funding was supplied to each site to cover the design of the trials and preparation for the implementation. All costs are reported, but the calculation of set-up costs excludes any costs that relate solely to the design or evaluation of the trials; these are referred to as trial-specific set-up costs.

As noted earlier, the trials were launched in May 2018. The recruitment and randomisation phase finished at the end of October 2019 and IPS service delivery to the treatment group finished in October 2020. Providers were asked to estimate staff time spent on trial-specific activities; for example, training on how to use the randomisation tool and to deliver the randomisation interviews, and time spent on activities related to IPS service delivery.^{[footnote 6]} This information was used to estimate trial-specific costs which were deducted from the total site costs to form an estimate of the costs of providing the IPS services to recruits in the treatment group.

The pre-delivery and delivery costs were collected for each site. However, to be able to consistently compare them with the benefits from Chapter 2, costs need to be allocated to the IW and OOW groups in SCR. The simplest method would be to proportionally allocate all costs based on the number of recruits receiving IPS in each trial group. This is the approach taken to allocate initial set-up costs. However, for the ongoing delivery costs it was expected that costs would vary between the IW and OOW groups. The SCR provider staff survey was used to estimate the proportion of time staff spent with their IW clients compared to their OOW clients. These proportions were then multiplied by the total IPS service delivery cost in Sheffield to obtain service delivery costs for each group.

The WHU was responsible for oversight of the IPS intervention and its staff were engaged in a range of activities including design, management of the trials, and their evaluation, and delivery support. Salary costs (including NI and pension contributions) were only calculated for WHU staff supporting the delivery of the IPS services. These costs were incurred across both sites rather than being site-specific and so were estimated for each trial group based on their share of the overall number of recruits receiving the IPS intervention.

The estimates of financial costs for each of the 2 sites are given in Section 3.3.1. WHU costs are set out in Section 3.3.2 and the 2 components are combined to calculate the total costs in Section 3.3.3. Both total costs and the cost per recruit in the treatment group are reported for each of the following groups: SCR IW, SCR OOW, WMCA and all OOW.

3.3.1 Site costs

As costs are site-specific, there is limited value in seeking to make comparisons across sites in expenditure on the different components; for example, in how much each spent on communications and marketing. However, the analysis does distinguish between set-up and ongoing running costs for the IPS service. The site costs are separated into 2 periods – pre-delivery period (up to May 2018) and the delivery period (May 2018 to October 2020). Costs that occurred after the trial delivery period ended are not considered as part of the economic evaluation.

SCR

Table 3.1 shows the costs incurred in the pre-delivery period for SCR and where possible separates trial-specific costs and site delivery costs. The first 2 rows of funding covering trial and evaluation design (£60,044 and £394,004) were documented in a Memorandum of Understanding. There was also set-up funding of £1,958,076. However, this funding cannot be disaggregated into trial-specific and site delivery-specific costs. It is assumed that it covered both trial-related and delivery-related set-up expenditure and so it is assigned equally to each of these.^{[footnote 7]} The responsiveness of total costs to varying this assumption is tested in a sensitivity analysis which can be found in Section 3.3.3.

Table 3.1 Set-up costs of IPS HLT in SCR

	Description	Trial-specific cost	Site delivery - specific cost
Trial design	Initial seed funding to be used in designing the trial	£60,044
Trial evaluation and design	Funding for ethics clearance, design process, delivery and collection of data for evaluation	£394,004
Set-up costs	Trial and intervention related costs	£979,038	£979,038
Total		£1,433,086	£979,038

Source: Economic evaluation spreadsheet

Table 3.2 shows the costs incurred during the delivery period of the trial. Ideally, it would be possible to exclude from the total site costs any costs associated purely with the running of the trial, as distinct from delivering the intervention. Within some of these categories of expenditure, such as management and administration, there might be non-operational costs that relate to trial delivery. Some of the other categories may also have included costs that related to both trial delivery and operational costs. However, it is not possible to remove them from the estimation, which may result in an overstatement of delivery costs. This concern can be partially addressed, however. Although the site financial data does not permit the separation of trial-specific versus intervention-specific costs, staffing levels are used to estimate the staffing cost associated with trial-related activities. This calculation involves multiplying total staffing costs by the proportion of time staff spend on trial-related activities.^{[footnote 8]} This is then deducted from the site financial costs as these costs were not relevant for delivering the intervention. The total intervention-specific site costs incurred in Sheffield are estimated to be £4,731,300 corresponding to a cost per participant of £1,547.

Table 3.2 SCR costs during the delivery period

Delivery period May 2018 to October 2020

Cost categories	Description	Cost
Project delivery	Includes the cost of the IPS service, CCG management fee and support for ethical processes in the clinical arena.	£4,221,829
Programme management and administration - external costs	Contracts within the HLT, data and information management services, data analysis, data presentation, supporting Local Integration Boards in each of the trial areas, and marketing and communication activity in the Sheffield CCG	£405,769
Programme management and administration - internal costs	Staffing costs, programme commissioning and contracting of trial activities	£549,715
Marketing, stakeholder participation and co-design	Providing marketing materials in a range of languages, social media marketing, printing promotional material and maintenance of the Working Win website	£409,500
Consultancy and temporary support	Programme design	£98,955
Business support	Support for finance and legal functions	£87,599
Contingency	Funds to help support unplanned spending	£34,777
Total cost		£5,808,146
Estimated staffing costs of trial related activities	Calculated using provider staffing costs adjusted for the proportion of time spent on conducting the randomisation interview and trial-specific training	£1,076,846
Total delivery-specific costs	Total site costs minus the staffing cost of randomisation and trial-specific training	£4,731,300
Number of participants	Number of participants in the treatment group	3,059
Cost per participant		£1,547

Source: Economic evaluation spreadsheet

In SCR the trial recruited 2 distinct groups of people: SCR IW and SCR OOW. Information from the staff survey was used to assess whether support to these trial groups varied, which would indicate delivery costs varied. The estimation required several pieces of information:

• for the SCR OOW the average proportion of recruits in the treatment group who had either substantial, moderate or few barriers to work

• for the SCR IW the average proportion of recruits in the treatment group who wanted to return to, or remain with, their existing employer, or find a new employer

• the frequency of meetings with recruits receiving IPS (face-to-face and virtual)

• the length of each meeting in minutes (face-to-face and virtual), reported in banded categories

Due to the pandemic, the delivery of the intervention shifted from face-to-face to virtual appointments and the survey allowed for the estimation of time spent on face-to-face and virtual appointments separately. The estimation took the category midpoint in terms of the length of each meeting to calculate the total number of minutes spent with each recruit receiving IPS in each trial group multiplied by the frequency of monthly meetings. The total number of minutes was then calculated, and the ratio of time spent on the SCR IW trial group versus the SCR OOW trial group was estimated.

This should be interpreted with some caution as the estimations rest on a survey that was completed by a small sample of SCR employment specialists and may not be representative of findings across all who delivered the IPS service. The results presented in Table 3.3 show that employment specialists spent three-fifths of their time on meetings with the SCR OOW trial group, compared with just two-fifths of the SCR IW trial group. The amount of time spent on virtual meetings was considerably lower than time spent on face-to-face meetings, but the proportion of time allocated to the SCR IW and SCR OOW groups was similar.

	SCR OOW	SCR IW
Notional monthly time spent on face-to-face appointments	329,722 minutes (5,495 hours)	189,610 minutes (3,160 hours)
% of monthly time allocated to each trial group (face-to-face)	63.5%	36.5%
Total cost	£3,004,376	£1,726,925
Number of recruits receiving IPS	1,799	1,260
Cost per recruit receiving IPS	£1,670	£1,371

Note: the proportion of time spent on virtual meetings was 62% for the SCR OOW and 38% for the SCR IW.

Source: Economic evaluation spreadsheet.

WMCA

Table 3.4 presents the initial funding in WMCA and attempts to separate the site delivery-specific costs from the trial-specific costs. The WMCA received an initial grant of £62,500 to ‘support [it] in the design and development of their’ trial which covered the period from September to December 2016. There was an additional grant for £33,000 in March 2017 ‘to support the implementation of the trial’. These amounts are well documented in the Memorandum of Understanding and costs can be easily categorised as trial-specific. However, as was the case in SCR, a later grant (£1,756,972) cannot be disaggregated into trial-specific and site delivery-specific costs. Again, a 50/50 split between trial-specific and site delivery-specific expenditure is assumed and varying this assumption is later tested in sensitivity analysis presented in Section 3.3.3.

Table 3.4 Set-up costs in WMCA

	Description	Trial-specific cost	Site delivery-specific cost
Trial design	Initial seed funding to be used in designing the trial	£62,500	-
Implementation phrase	Funding for ethics clearance, design process, delivery and collection of data for evaluation, appointing providers	£33,000	-
Set-up costs	Trial and intervention related costs	£878,486	£878,486
Total		£973,986	£878,486

Source: Economic evaluation spreadsheet

Table 3.5 shows various categories of spending in WMCA. Provider costs made up the vast majority of spending at £5.5 million. As before, estimated trial staffing costs relating to randomisation interviews and trial-specific training were deducted from the site financials provided by the WHU. Thus, the total cost for WMCA came to £5,490,282.

Table 3.5 Costs during the delivery period in WMCA

Delivery period May 2008 to October 2020

Cost categories	Description	Cost
Providers	Individual provider costs which have been grouped together	£5,536,269
Programme team	Staffing levels of the programme team	£710,420
CCG support costs	Includes programme team expenses, communications, IT, legal support and contingency	£66,920
Third party	MI system development, performance management, marketing, procurement and implementation support	£732,620
Total trial cost		£7,046,229
Provider VAT costs	Some providers had to pay VAT, which is removed to focus on real resource costs	£691,000
Estimated staffing costs of trial-specific activities	Calculated using provider staffing costs adjusted for the proportion of time spent on conducting the randomisation interview and trial-specific training[footnote 9]	£864,947
Total cost (excluding trial-specific costs)		£5,490,282
Number of recruits receiving treatment	Number of recruits in the treatment group	1,837
Cost per recruit receiving treatment		£2,989

Source: Economic evaluation spreadsheet

3.3.2 Work and Health Unit costs

The WHU provided a list of staff working on the trials, their role, grade, start and end dates and the estimated percentage of full-time equivalent time spent working on service delivery as opposed to other tasks.^{[footnote 10]} This list included staff from the DWP, NHS-E and DHSC. It was used to estimate the staff costs for those working on the delivery of the IPS services. Firstly, those who worked solely on trial design, or the evaluation of the trial, were excluded. Then salary mid-points of relevant pay scales were used to estimate the salary costs of the relevant staff. This calculation took into account the location of the employee so that the London weighting could be applied where relevant. Employer-specific pension contributions and NI employer contributions were added to calculate final salary costs.

Because there were no new offices for the trials, it was assumed that there were no additional overhead costs in addition to the NI contributions. The analysis tests the sensitivity of the findings to this assumption which is discussed in more detail in Section 3.3.3.

The total estimated WHU delivery staff costs were £1,189,601 in the delivery period between May 2018 and October 2020. WHU costs were also incurred in the pre-delivery period. The staffing costs in the pre-delivery period were £894,213 meaning the total WHU cost comes to £2,083,814.

3.3.3 Total costs

The total cost of the delivery of IPS services across both sites, comprised of site costs (initial set-up and delivery costs) and WHU costs, was £14.2 million. For comparison with the benefits in Chapter 2 it is important to break down the costs by site and for the 2 trial groups in SCR (SCR IW and SCR OOW). Table 3.6 provides the results of this breakdown. This assumes the initial set-up costs were split by the proportion of recruits receiving IPS in each trial group. The same logic was applied to the WHU costs in SCR which were shared proportionally between the 2 trial groups. Though SCR and WMCA had similar total costs associated with IPS delivery, the cost per recruit receiving IPS was considerably higher in WMCA owing to the smaller number of individuals recruited to the trial.

Table 3.6 Estimates of total costs across SCR and WMCA

	All SCR	SCR IW	SCR OOW	WMCA	All OOW
Initial set-up costs (excluding any grants used for designing the trial)	£979,038	£403,265	£575,773	£878,486	£1,454,259
Site costs	£4,731,300	£1,726,925	£3,004,376	£5,490,282	£8,494,658
WHU costs	£1,301,958	£536,276	£765,682	£781,856	£1,547,538
Total cost	£7,012,297	£2,666,465	£4,345,831	£7,150,624	£11,496,455
Number of recruits receiving IPS	3,059	1,260	1,799	1,837	3,636
Cost per recruit receiving IPS	£2,292	£2,116	£2,416	£3,893	£3,162

Source: Economic evaluation spreadsheet

The total delivery-specific costs are calculated from the data held in documents such as memoranda of understanding, grant agreements and contracts, so it is likely they provide a reasonable estimate of the cost of setting up and delivering the IPS service, though they may not be exact. Table 3.7 shows the percentage change in costs when varying several of the assumptions to illustrate how sensitive the estimates of costs are to changes in the underlying assumptions.

For WHU costs no overhead costs were assumed since there were no additional DWP or NHS-E offices needed for the IPS service. In Panel A this is changed to assume overheads account for 30% of staff costs, rather than including calculated employer NI and pension contributions, as was the case in Table 3.6.^{[footnote 11]} Table 3.7 shows that using the 30% overhead rate for WHU costs decreases total costs by around 0.5% for each group. This indicates that actual NI and pension contributions for WHU staff were greater than 30% of salary costs.

Secondly, WHU costs of staff working on the delivery of the IPS service were previously assumed to be incurred across the duration of the trial period. This assumption is relaxed in Panel B so that only the costs of employing WHU delivery staff during the delivery period are included. Again, this leads to a reduction in total costs by between 4% and 8%.

Finally, the lack of clarity on the share of initial funding given to sites which was used for trial-specific or delivery-specific activities led to the assumption of a 50/50 split in Section 3.3.1. Panel C assumes 40% of set-up costs were related to delivery of the IPS services, whilst Panel D assumes 75%. These figures were chosen arbitrarily and may be varied accordingly. Table 3.7 shows that when assuming 40% of the initial set-up costs were related to setting up the service (Panel C), total costs decreased by between 2% and 4%. In Panel D, with the more conservative assumption of assigning 75% of costs as delivery-specific, total costs increase by between 4% and 8%.

Table 3.7 Sensitivity of costs to varying assumptions

	All SCR	SCR IW	SCR OOW	WMCA	All OOW
Total cost	£7,012,297	£2,666,465	£4,345,831	£7,150,624	£11,496,455

Overhead cost for WHU	All SCR	SCR IW	SCR OOW	WMCA	All OOW
New cost	£6,982,391	£2,654,148	£4,328,244	£7,132,665	£11,460,909
% change	-0.4%	-0.5%	-0.4%	-0.3%	-0.3%

WHU cost	All SCR	SCR IW	SCR OOW	WMCA	All OOW
New cost	£6,430,411	£2,553,665	£4,003,624	£6,801,188	£10,804,811
% change	-8%	-4%	-8%	-5%	-6%

40% of initial set-up cost spent on delivery	All SCR	SCR IW	SCR OOW	WMCA	All OOW
New cost	£6,816,489	£2,585,812	£4,230,677	£6,974,926	£11,067,597
% change	-3%	-3%	-3%	-2%	-4%

75% of initial set-up cost spent on delivery	All SCR	SCR IW	SCR OOW	WMCA	All OOW
New cost	£7,501,816	£2,868,098	£4,633,718	£7,589,867	£11,964,822
% change	7%	8%	7%	6%	4%

Source: Economic evaluation spreadsheet

Ratio of benefits to costs

This chapter combines the benefit estimates from Chapter 2 with the cost estimates from Chapter 3. Costs and benefits are aggregated and reported as a benefits cost ratio (BCR) for both the financial and economic case.

4.1 Introduction

For any intervention where benefits exceed costs, the BCR will be greater than 1, indicating that the intervention provides value for money. A BCR of less than 1 indicates that the costs of the intervention exceed the estimated monetary benefits. The following 2 sections report the BCRs for the SCR IW and the all OOW trial groups before showing how the BCR for the all OOW group compares between the SCR OOW and WMCA trial groups. In addition to reporting the BCR for the HLT, this chapter discusses how the results might be affected by any costs and benefits that could not be monetised, any major caveats to the main findings, and the results of various sensitivity tests.

4.2 Financial case

As reported in Chapter 2, the IPS services had few clear financial benefits. Table 4.1 shows the net present value (NPV) and BCR for the SCR IW and the pooled (all OOW) trial groups. This provides an overview of the financial case for the IPS service across IW and OOW recruits to the treatment group. The NPV is the present value of benefits minus the present value of costs over the 3-year period following randomisation and reflects the net impact of the IPS services. Results are presented for each of the reference, optimistic and pessimistic scenarios as defined in Chapter 2.

Table 4.1 and Table 4.2 reflect the fact that whilst positive employment impacts were seen for the SCR IW and WMCA trial groups, the financial benefits were much smaller than costs of delivering the IPS service. Even in the optimistic scenario the BCR was only 0.02 for the SCR IW and the WMCA trial groups, whilst in the pessimistic case, the BCR was 0.01. This implies for every £1 spent on the IPS service, it delivered between £0.01 and £0.02 of financial benefits to the exchequer for these trial groups. For the all OOW and SCR OOW trial groups, where there was no discernible impact from the IPS service on employment, the negative NPV was equivalent to the discounted costs of running the IPS service.

Table 4.1 Total net present value and benefit to cost ratios for the SCR IW and all OOW trial groups

Scenario	Net present value (SCR IW)	Benefit to cost ratio (SCR IW)	Net present value (all OOW)	Benefit to cost ratio (all OOW)
Reference	-£2.6 million	0.02	-£11 million	0
Optimistic	-£2.5 million	0.02	-£11 million	0
Pessimistic	-£2.6 million	0.01	-£11 million	0

Source: Economic evaluation spreadsheet

Table 4.2 Total net present value and benefit to cost ratios for the SCR OOW and WMCA trial groups

Scenario	Net present value (SCR OOW)	Benefit to cost ratio (SCR OOW)	Net present value (WMCA)	Benefit to cost ratio (WMCA)
Reference	-£4.2 million	0	-£6.7 million	0.01
Optimistic	-£4.2 million	0	-£6.7 million	0.02
Pessimistic	-£4.2 million	0	-£6.7 million	0.01

Source: Economic evaluation spreadsheet

4.3 Economic case

Although there were few financial benefits of the IPS services, some economic benefits were found. Table 4.3 and Table 4.4 present the NPV and BCR for the economic case for the trial groups. Under the reference scenario, Table 4.3 shows that both the SCR IW and all OOW trial groups had BCRs greater than 1. The public benefit was estimated to be £3.4 million for the SCR IW trial group and £2.4 million for the all OOW trial groups. This suggests that every £1 spent on the intervention delivered £2.32 of benefits for the SCR IW trial group, and £1.22 of benefits for the all OOW trial groups respectively.

The BCR is much lower for the all OOW trial group largely because there were no discernible health impacts in WMCA whereas for both SCR trial groups small positive impacts on health and wellbeing were observed that were statistically significant at the 99% level. As these make a sizeable contribution to economic benefits, the lack of health impacts in WMCA reduce the BCR for the all OOW group. The pessimistic scenario shows a positive BCR for the SCR IW group. This makes the economic case for the intervention compelling for the SCR IW group. It is less obvious for the all OOW group, as in the pessimistic scenario the BCR is below 1, suggesting that for the OOW group the costs of the IPS service exceed the benefits to society.

Table 4.3 Total net present value and BCRs for the SCR IW and all OOW trial groups

Scenario	Net present value (SCR IW)	Benefit to cost ratio (SCR IW)	Net present value (all OOW)	Benefit to cost ratio (all OOW)
Reference	£3.4 million	2.32	£2.4 million	1.22
Optimistic	£8.3 million	4.18	£14.6 million	2.32
Pessimistic	£700,000	1.26	-£4 million	0.64

Source: Economic evaluation spreadsheet.

Table 4.4 explores how the BCR for the all OOW trial group varies by site and illustrates how the lack of health impacts in WMCA affect the BCR. For SCR OOW the benefits to society from every £1 spent on the IPS service range from £1.03 in the pessimistic scenario to £3.77 in the optimistic scenario. The central estimate is that £1 spent on the IPS service for the SCR OOW group yields £2.02 in benefits. In WMCA the BCR is less than 1 in all the scenarios considered, indicating that the costs to society exceed the benefits.

Table 4.4 Total net present value and benefit to cost ratios for the SCR OOW and WMCA trial groups

Scenario	Net present value (SCR OOW)	Benefit to cost ratio (SCR OOW)	Net present value (WMCA)	Benefit to cost ratio (WMCA)
Reference	£4.3 million	2.02	-£6.8 million	0.00
Optimistic	£11.7 million	3.77	£5.4 million	0.20
Pessimistic	£120,000	1.03	-£6.8 million	0.00

Source: Economic evaluation spreadsheet

Figure 4.1 to Figure 4.4 show how much each of the different components of costs and benefits contribute to the economic case for the IPS service for each of the trial groups. They also report overall net benefits (benefits minus costs) for the reference scenario. Appendix E shows the components of costs and benefits for the optimistic and pessimistic scenarios. Figure 4.1 shows that for the SCR IW trial group, which had a BCR of 2.32 in the reference scenario, the entirety of this positive effect came from the improvements in HRQOL resulting from the IPS service. For the all OOW trial group (Figure 4.2) there were positive health impacts which, when monetised, were greater than the cost of delivering the intervention. This was also the case for the SCR OOW group (Figure 4.3).

Figure 4.1 Components of the economic case for SCR IW trial group, reference scenario

Reduced healthcare costs	Increased childcare costs	Increased travel costs	Improved health-related quality of life	Cost	Net benefit
£51,526	-£13,547	-£31,516	£6,044,873	£2,603,396	£3,447,940

Source: Economic evaluation spreadsheet

Figure 4.2 Components of the economic case for all OOW trial group

Improved health-related quality of life	Cost	Net benefit
£13,413,916	£11,036,181.46	£2,377,734

Source: Economic evaluation spreadsheet

Figure 4.3 Components of the economic case for SCR OOW trial group

Improved health-related quality of life	Cost	Net benefit
£8,544,180	£4,235,571	£4,308,609

Source: Economic evaluation spreadsheet

Figure 4.4 shows that in the WMCA the lack of a positive economic case for the IPS service was driven by the fact that there were few positive impacts, resulting in few benefits relative to the costs of running the service. Increased travel costs and childcare costs for participants who entered work had a negative impact on the BCR, but were small compared with delivery costs.

Figure 4.4 Components of the economic case for the WMCA trial group

Increased childcare costs	Increased travel costs	Reduced healthcare costs	Cost	Net benefit
-£21,956	-£55,838	£92,376	£6,800,610	£6,786,028

Source: Economic evaluation spreadsheet

Omitted costs and benefits and potential impact on findings

As noted in Chapter 2, only some of the impacts expected to result from the IPS services can be estimated or monetised. This section reviews the potential omissions and considers how each might affect the findings of the economic evaluation in terms of the benefit to costs ratio. It focuses on potential costs and benefits that are entirely excluded from the analysis, rather than those which may be under or over estimated. Section 4.5 considers how varying the assumptions made in the analysis might affect the main findings.

The omitted costs and benefits largely fall on recruits, employers and society more generally, rather than the exchequer. Whilst receiving support from the IPS service, recruits in the treatment group may have incurred some costs in terms of the time that they spent meeting with employment specialists and engaging with other services and activities that they were referred to as part of the support they were offered. They may also have incurred some travel costs due to their participation, for example, to travel to meetings with their employment specialists in community settings. In common with other studies, the cost-benefit analysis does not seek to estimate the costs of taking part for the treatment group. This exclusion is likely to inflate the economic BCR.

Recruits in the treatment group who entered employment as a result of receiving IPS may have incurred some work-related expenditure in addition to commuting and childcare costs which are valued in the economic evaluation. For example, they may have needed to purchase new clothes or equipment. These expenses were likely to vary depending on the nature of their job and how long they had been out of work, and so would be difficult to estimate but, averaged across recruits to the treatment group who entered employment due to IPS, would be likely to be small relative to the potential benefits. However, taking these expenses into account would be likely to reduce the economic BCR. Set against this, those who entered work as a result of IPS may have received some non-financial benefits, even in the absence of earnings impacts (none were observed in either of the trials). Again, these will vary between employees and would be difficult to measure or value, but in this case the BCR would probably be higher if these benefits were included.

As well as omitted costs and benefits for those in the treatment group who entered employment, there are potential costs to society from the additional environmental costs of pollution from an increase in commuting. Including these environmental costs and their impact on health would be likely to reduce the BCR.

Whilst the impact study found that the IPS services did not result in increased earnings for recruits to the treatment group who entered work, it may have raised productivity if employers took any increase in output as higher profits, rather than paying higher wages. As it was not possible to directly measure the impact of the IPS services on productivity, it is possible that the BCR understates the benefits of the services.

As mentioned previously, the analysis only considers the benefits that result from receiving the IPS services over a 3-year period. The positive effects on health may result in longer-term savings which are not captured in the BCR. Indeed, more generally, it seems likely that at least some impacts will continue beyond the 3-year period considered in the analysis, whereas no further delivery costs would be incurred related to the treatment group. This means that the BCR is likely to be an underestimate of the true benefits resulting from expenditure on the IPS services.

Table 4.5 Expected impact of including the omitted benefits and costs on the Economic BCR

Omitted benefit or cost	Likely impact of inclusion in economic BCR
Time participants spent on IPS activities	Decreased
Other work-related expenditure	Decreased
Non-financial employment benefits	Increased
Environmental costs of commuting	Decreased
Potential productivity gains	Increased
Longer-term impacts	Increased

Source: Economic evaluation spreadsheet

4.5 Other caveats

The impact report noted that there was evidence of differences between the SCR IW treatment and control groups which may affect the robustness of findings from the final survey. More specifically the positive effects on health and wellbeing for the SCR IW trial group, which are based on analysis of survey data, may not be fully reliable (for the reasons described in section 2.4.7), although consistency with the findings for the SCR OOW trial group provides some reassurance in this regard. As the health effects make a positive contribution to the economic BCR in particular, if the health and wellbeing impacts seen for the SCR IW trial group were less than estimated, this could reduce the economic BCR considerably.

Appendix B provides a detailed discussion of the extent to which the analysis is likely to capture the full range of expected impacts from the IPS service. In some cases, the monetary value used to estimate costs and benefits, such as commuting or childcare costs, are based on national averages. If these values were not typical of the costs or benefits incurred by the treatment group, the BCR may be under or overstated. If the urban nature of the trial sites meant that travel costs were lower than the national average, the use of national averages might result in the BCR being understated.

The estimate of how employment might affect childcare costs does take into account possible differences in the use of paid childcare by lone parents and couples, but the composition of households is only observed at the time of randomisation. The BCR does not therefore reflect any changes in the composition of households over time. However, whilst the use of childcare by individual households may change over time as family circumstances change, there is no reason to expect that across the treatment group changes in household composition over a 3-year period would have a sizeable effect on the costs of childcare. It is therefore unlikely that basing the estimates of childcare costs on family circumstances at baseline would have a clear effect on the BCR.

The monetary values used to estimate total costs assume that there is no fixed component, so all costs vary in a linear way with impacts from receiving the IPS services. In reality, some costs may have a fixed component, however. For example, someone paid to work half a day may incur the same costs of travelling to work as someone who is paid for a full working day. Assuming that costs associated with working are more variable than they are likely to be in practice would be expected to inflate the BCR.

The analysis assumes that the employment impacts resulting from IPS also have an impact on the use of health services, in line with DWP’s social cost-benefit analysis framework. In valuing the savings resulting from reduced healthcare costs, the economic evaluation takes into account the proportion of the treatment group who were claiming ESA in each year following randomisation and assumes a greater saving from reduced healthcare costs from ESA recipients entering work. Again, the assumptions about healthcare savings are based on national averages. Given that the HLT IPS services were targeted at those with health problems, the healthcare savings from assisting recruits into employment may be greater than the national average even for those recruits who were not on ESA at any point in the 3-year period considered when calculating the BCR. This would potentially mean that the BCR underestimates the healthcare savings resulting from the IPS services.

The risk of double counting meant that it was not considered advisable to seek to value the estimated impact of the IPS service on both general health and on wellbeing, seen in SCR. However, the measure used to capture general health effects (EQ5D5L) only included one item on mental health and it is possible that it undervalues the positive impact on wellbeing from the IPS service. For example, any wellbeing effects of receiving IPS might result in positive benefits for families, friends, colleagues, employers, and society, as well as recruits themselves. In not seeking to value the wellbeing effects of the IPS service in SCR, it is possible that the BCR is underestimated.

Finally, it is important to note, in common with other economic evaluations, that the accuracy of the estimated BCRs depends in part on how well the inflation rates used reflect real values for the particular costs and benefits under consideration at a given point in time. Whilst the analysis largely uses historic CPI rates, it is necessary to forecast figures for the 2021 to 2022 financial year and these are therefore subject to a greater degree of uncertainty than the CPI figures for earlier years. It is also possible that CPI does not capture the true inflation rate for every cost and benefit valued in the analysis.

4.6 Statistical certainty and sensitivity analysis

The cost-benefit analysis presented in this report is based on statistically significant results from the HLT 12-month impact study. Whilst these impact estimates met accepted thresholds for statistical significance, they are still subject to statistical uncertainty. This means that the BCRs set out in this chapter are also subject to statistical uncertainty.

The uncertainty surrounding these estimates of the BCRs cannot be expressed using conventional statistical tools such as hypothesis testing. Instead, in common with other studies, Monte Carlo probabilistic sensitivity analysis is undertaken to provide an insight into the confidence with which the findings can be interpreted (Ministry of Housing, Communities and Local Government 2019). As the BCRs were very low for the financial case, even for the SCR IW and WMCA trial groups where positive employment impacts were seen, this section focuses on the economic case for the IPS service.

The sensitivity analysis is carried out by randomly selecting values from the probability distributions of each impact estimate, based on their standard errors. These distributions are then used to simulate the distributions of the BCRs for each trial group. This process of randomly selecting values from the probability distribution is performed 10,000 times, across the reference, optimistic and pessimistic scenarios for the economic BCRs. This provides an estimate of the probability that the BCR is lower than 1; in other words, that the IPS service costs more to deliver than the value of the benefits it produces. In addition to this, a number of other measures are calculated:

• 95% confidence intervals for the estimated BCR, which means it is 95% certain that the true BCR lies within this range.

• The estimated BCR under the assumption that the impact estimate is at either the upper or lower limit of its 95% confidence interval. Again, there is 95% certainty that the true impact of the IPS service is within this range and so these measures show how the BCR might vary if the impact estimate was at either end of these extremes. For the SCR IW trial group where both the health and employment impacts are simulated, when estimating the BCR one impact is kept at its central estimate whilst the other is assumed to be either at the upper or lower limit of its 95% confidence interval.

• The impact estimate required to break even.

For the SCR IW, SCR OOW and the all OOW trial groups the health impact was used to produce the Monte Carlo probabilistic sensitivity analysis for the economic case. In the case of WMCA, the 12-month employment impact was used as the basis for the simulation.

The results of the sensitivity analysis are reported in Table 4.6 to Table 4.9. The 95% confidence intervals around the BCRs mean it is possible to say with a high degree of certainty that the trial groups which have economic BCRs above 1 in their reference scenarios (SCR IW, SCR OOW and all OOW) are indeed likely to have a BCR above 1. This is also the case in the optimistic scenario, as well as in the pessimistic scenario for the SCR IW and SCR OOW trial groups. In the case of the all OOW trial group, the 95% confidence intervals around the BCR are both below 1. In this case, if the assumptions underlying the pessimistic scenario are accurate, it seems unlikely that the monetary benefits of the IPS service outweigh the costs.

Table 4.6 Results from Monte Carlo probabilistic sensitivity analysis: SCR IW economic case simulating the health impact

	Reference	Optimistic	Pessimistic
Central estimate of BCR	2.32	4.18	1.27
95% confidence interval of BCR	2.30 to 2.34	4.14 to 4.22	1.26 to 1.28
Probability of not achieving break-even	12.98%	5.43%	32.60%
BCR for upper limit of 95% CI for impact estimate	4.21	7.70	2.13
BCR for lower limit of 95% CI for impact estimate	0.04	0.26	0.02
Break-even impact	0.01	0.01	0.02

Source: Economic evaluation spreadsheet

Table 4.7 Results from Monte Carlo probabilistic sensitivity analysis: All OOW economic case simulating the health impact

	Reference	Optimistic	Pessimistic
Central estimate of BCR	1.22	2.32	0.64
95% confidence interval of BCR	1.21 to 1.22	2.31 to 2.34	0.63 to 0.64
Probability of not achieving break-even	30.44%	5.51%	97.05%
BCR for upper limit of 95% CI for impact estimate	2.05	3.87	1.02
BCR for lower limit of 95% CI for impact estimate	0.27	0.66	0.15
Break-even impact	0.02	0.01	0.04

Source: Economic evaluation spreadsheet

Table 4.8 Results from Monte Carlo probabilistic sensitivity analysis: SCR OOW economic case simulating the health impact

	Reference	Optimistic	Pessimistic
Central estimate of BCR	2.02	3.77	1.03
95% confidence interval of BCR	2.00 to 2.03	3.75 to 3.80	1.02 to 1.04
Probability of not achieving break-even	10.09%	3.57%	47.02%
BCR for upper limit of 95% CI for impact estimate	3.54	6.60	1.72
BCR for lower limit of 95% CI for impact estimate	0.35	0.79	0.19
Break-even impact	0.01	0.01	0.03

Source: Economic evaluation spreadsheet

Table 4.9 Results from Monte Carlo probabilistic sensitivity analysis: WMCA economic case simulating the employment impact

	Reference	Optimistic	Pessimistic
Central estimate of BCR	0.00	0.20	0.00
95% confidence interval of BCR	0.00 to 0.00	0.2 to 0.2	0.00 to 0.00
Probability of not achieving break-even	100.00%	100.00%	100.00%
BCR for upper limit of 95% CI for impact estimate	0.00	0.21	0.00
BCR for lower limit of 95% CI for impact estimate	0.00	0.21	0.00
Break-even impact	-	-	-

Source: Economic evaluation spreadsheet

The probability of not achieving break-even in the reference scenario for the economic case for the SCR IW, SCR OOW and all OOW trial groups is 12.66%, 10.09% and 30.44% respectively. Naturally, the optimistic scenarios suggest that the probability of not breaking even is lower than in the reference cases. In the pessimistic scenario, the likelihood of reaching break-even is much lower, and in the case of the all OOW trial group the probability of not reaching break-even is 97.05%.

Even for the trial groups where the central estimate of the BCR for the economic case was greater than 1 across all 3 scenarios (the SCR IW and SCR OOW trial groups), the BCRs produced by using the upper and lower limits of the 95% confidence intervals of the impact estimates fell either side of 1. Given the statistical uncertainty surrounding the central impact estimates, it is not possible to say with confidence whether the BCRs would have exceeded 1 across any of the scenarios. The degree of statistical uncertainty surrounding the BCRs for the SCR IW, SCR OOW and all OOW trial groups mean that if the interventions were replicated, it is possible that the positive net economic benefits estimated in this study would not be apparent.

For the economic case for the WMCA trial group, the IPS service was highly unlikely to achieve break-even across any of the 3 scenarios. As the impacts required to achieve break-even were outside of the possible range of values for the outcome measure, this is not reported. All statistics computed as part of the sensitivity analysis indicated that it is possible to say with a high degree of certainty that the IPS service for the WMCA trial group failed to achieve break-even.

5. Summary and conclusions

The cost-benefit analysis examined the relative benefits and costs of the IPS services across the 2 trial sites, based on the impact estimates from the randomised controlled trial. This chapter summarises the main findings of the economic evaluation, covering the financial and economic case for the IPS services. It considers limitations of the analysis and conclusions that can be drawn, given the likely robustness of findings.

5.1 Summary

The purpose of the economic evaluation was to consider whether the results from the impact evaluation and specifically, improved health for the SCR IW and SCR OOW trial groups, and a positive impact on the primary employment outcome for the SCR IW and WMCA trial groups, meant that overall the IPS services offered value for money for the exchequer and society within 3 years of randomisation. Whilst the IPS services did produce some financial benefits for the SCR IW and WMCA trial groups, the lack of earnings effects meant that these were insufficient to offset the costs of running the services. In the financial case, the benefit to cost ratios (BCRs) were zero or close to zero for all trial groups.

The economic case for the IPS services was stronger, mainly due to the impact that it had on HRQOL in SCR. The main analysis suggested that for the SCR IW and the SCR OOW trial groups receiving IPS did provide value for money when considering the economic case even under the most pessimistic scenario. Although the BCR was greater for the SCR IW group than the SCR OOW group, this finding should be treated with caution, given differences in survey response rates between the treatment and control groups for the SCR IW trial group. As the IPS service had no impact on the primary health outcome in WMCA, the BCR for the economic case was zero or close to zero under all scenarios. This meant that when the OOW group in SCR was combined with WMCA trial group (all of whom were OOW), the economic benefits from the IPS services only outweighed the costs under the reference and optimistic scenarios.

In addition to exploring how the BCRs varied when making more or less optimistic assumptions about expected costs and benefits, further analysis was carried out to assess the likely robustness of the findings. The probability of not achieving break-even for society was around one-third or more under the pessimistic scenario in all cases. The lowest probability of not achieving break-even under the reference scenario was seen for the SCR OOW trial group, where it was 10.1%. For the SCR IW group the probability of not achieving break-even was 13.0%. The sensitivity 60 testing also explored how the BCR might vary if the impact estimates were towards the lower end of expectations, based on the degree of imprecision around the central estimates of impact. This analysis found that if the study was repeated, there is a possibility that the economic benefits of the IPS services would not outweigh the costs in SCR even in the most optimistic scenario. In summary, the analysis of statistical certainty in the findings suggests that the estimated BCRs should be treated with a high degree of caution.

5.2 Limitations of the analysis

This section summarises the main limitations of the analysis and their implications for the overall conclusions, as well as how they could affect the potential BCRs. Firstly, it is apparent that there are some ambiguities in the available data on the costs of operating the IPS services. Some of the budget for the HLT was spent on evaluating the impact of the IPS services, rather than delivering the services to the treatment group. The CBA has sought to exclude any costs which were only incurred as part of the evaluation of the IPS services, but to do this it has been necessary to make a series of assumptions about components of costs to include and exclude, as well as salary costs for staff in particular job roles and grades, and hours worked where this was not recorded. It was also necessary to make assumptions about the portion of the estimated delivery-specific costs spent on the SCR IW and SCR OOW trial groups, based on analysis of a staff survey. This relied on recall, rather than information recorded as each interaction with recruits to the treatment group took place, and so only provides a rough indication of the likely costs of delivering the IPS service to the SCR IW and OOW trial groups.

More generally, the accuracy of the BCRs depend on whether the assumptions underlying the analysis reflect reality. The results reported in Chapter 4 include a consideration of how the BCRs are likely to vary when the assumptions underlying the main analysis are adjusted to allow for more optimistic or pessimistic scenarios. Whilst in the case of SCR in particular it is apparent that economic benefits are likely to outweigh the costs of the IPS service under all of the scenarios, it is clear that the BCR differs substantially depending on whether the assumptions made are more optimistic or more pessimistic.

As it was not possible to obtain access to NHS Digital data to estimate the impact of the IPS services on the use of health services, it was instead necessary to make assumptions about how the employment impact observed for the SCR IW and WMCA trial groups might affect the use of health services. These assumptions were based on the DWP’s social cost-benefit analysis framework as well as the proportion of recruits to the treatment group in receipt of ESA. However, there is some uncertainty over whether the potential savings resulting from any reduction in the use of healthcare by those who entered work as a result of the IPS services are accurately reflected in the CBA. This is also the case for the wellbeing impacts seen in SCR. As there is no established way of valuing wellbeing alongside the impacts on the more general HRQOL measure, it is possible that some of the potential benefits of the IPS services are not monetised in the CBA, resulting in the economic BCR being underestimated.

A further limitation of the analysis is the degree of uncertainty surrounding the impact estimates. As noted, the sensitivity testing demonstrated that if the impact estimates were at the lower end of expectations, the costs of delivering the IPS services would exceed the economic benefits, even in SCR where the BCR was highest. For the SCR IW trial group, there is also some uncertainty about whether the health impacts seen were affected by differential response rates for the treatment and control groups to the 12-month survey. Again, access to NHS Digital Data would have made it possible to explore whether there were reductions in the use of health services as a result of IPS receipt. As these data would have been available for most recruits to the trial this would have provided another means of assessing the likely robustness of the health impacts seen for the SCR IW group.

As the impact evaluation only sought to estimate the impact of the IPS services over the 12 months following randomisation, it is uncertain how the estimated impacts might evolve over a longer period of time. For those outcomes that could be observed in administrative data sources, that is, mainly those related to employment, it was possible to estimate impacts over a longer timeframe to inform the assumptions about the potential benefits that might be experienced over a 3-year period. However, this was not possible for impact estimates derived from survey data, (the health and wellbeing outcomes relied upon survey data), and so it is more difficult to know whether the assumptions made on the evolution of health effects are likely to be valid.

As noted previously, the monetary values used to estimate costs and benefits are based on national averages and so it is possible that these do not accurately reflect circumstances in SCR or WMCA. Whilst potentially this may result in the BCR being underestimated, in some cases there may be a fixed component to costs that is not taken into account in the calculations. In this case, the BCR may be overstated. As with any economic evaluation, there is some uncertainty over whether the inflation rates used accurately capture inflation for particular costs and benefits and so it is unclear how this may affect the actual BCRs.

Finally, some of the likely costs and benefits of the IPS service are difficult to observe or estimate. This is particularly evident when seeking to estimate the economic BCR. It is possible that the BCR underestimates the economic benefits of the IPS services if there are productivity gains to employers which are taken as increased profits, rather than increased earnings for recruits to the treatment group. Also, recruits are likely to experience some costs of participation and the exclusion of these costs in the current study may result in the economic BCR being overstated. There may be some environmental and health costs to society from increased travel to IPS meetings or work. In addition, those who enter employment may incur some work-related expenditure beyond the commuting and childcare costs which are included in the calculation of BCRs. However, those entering work may gain some non-financial benefits which could offset at least part of any additional work-related expenditure, making it difficult to predict whether these exclusions might result in the economic BCR being over, or underestimated.

5.3 Conclusions

In conclusion, the economic evaluation of the HLT found that over a 3-year period, there was some evidence to suggest that the economic benefits that the IPS service in SCR produced exceeded the costs of delivery. This was driven by the impact that the IPS service appeared to have on the primary health outcome for both the IW and OOW groups in SCR. By contrast, no health impact was evident in WMCA and so the economic evaluation found that there was no economic case for the IPS service as delivered in WMCA.

Considering benefits over a longer time period would be likely to strengthen the economic case for the IPS services. However, the uncertainty surrounding the impact estimates means that even in SCR there is a risk that the costs of running the IPS service might exceed the economic benefits if the trial were to be repeated. The costs to the exchequer of funding the IPS services are greater than the likely financial benefits across both trial sites. Whilst the IPS services may produce a net benefit to society in some circumstances, it is apparent from the differences in the outcome measures affected in each site that local variation in how the service operates and the relative weight attached to health or employment outcomes has an important bearing on the ratio of benefits to costs.

The impact study was only able to consider impacts 12 months after randomisation for the full cohort of recruits to the treatment group and across all outcome measures. This creates a degree of uncertainty about how impacts might change over time and how this might affect the BCRs. As a result, it was decided to focus on estimating the BCR over a period of 3 years after randomisation. If it were possible to gain access to the NHS Digital data, or even later extracts of DWP and HMRC data, longer-term impacts could be estimated. This would be likely to reduce variation in the BCRs between each of the 3 scenarios considered. It may also be feasible to estimate the BCRs over a longer period of time.

Bibliography

Devlin, Nancy J, Koonal K Shah, Yan Feng, Brendan Mulhern, and Ben van Hout. 2018. (Valuing health‐related quality of life: An EQ‐5D‐5L value set for England - Devlin - 2018 - Health Economics - Wiley Online Library

Fujiwara, Daniel. 2010. ‘The Department for Work and Pensions Social Cost-Benefit Analysis Framework’. Working Paper 86. Department for Work and Pensions.

HM Treasury. 2020. The Green Book: Central Government Guidance on Appraisal and Evaluation’

‘GDP Deflators at Market Prices, and Money GDP December 2021 (Quarterly National Accounts)’. GDP deflators at market prices, and money GDP December 2021 (Quarterly National Accounts) - GOV.UK (www.gov.uk)

HM Treasury, and Welsh Government. 2018. ‘Guide to Developing the Programme Business Case. Better Business Cases: For Better Outcomes’.

Jahoda, M. 1982. Employment and Unemployment: A Social-Psychological Analysis. Cambridge: Cambridge University Press.

National Institute for Health and Care Excellence. 2012. Methods for the development of NICE public health guidance (third edition)

Office for Budget Responsibility. 2021. Economic and fiscal outlook - October 2021 - Office for Budget Responsibility (obr.uk)

Office for National Statistics. 2022.Consumer price inflation tables - Office for National Statistics

Appendix A – Assumptions underlying the reference, optimistic and pessimistic scenarios

Table A. 1 Assumptions for the reference, optimistic and pessimistic scenarios

Input	Reference scenario	Optimistic scenario	Pessimistic scenario
Employment related discount rate	3.5%	3.0%	4.0%
QALY discount rate	1.5%	1.5%	3.5%
2021 to 2022 CPI inflation forecast	3.3%	3.8%	2.8%
QALY value	Fixed	GDP deflator applied	Fixed
Substitution effects	0%	0%	20%
Social cost of Exchequer Finance	0%	20%	0%
Employment impact trajectory	Year 2 impact = 22-month impact scaled up to 24-months linearly Year 3 impact = Year 2 impact	Year 2 impact = 22-month impact scaled up to 24-months linearly Year 3 impact = upward trend from Year 1 to Year 2 continues	Year 2 impact = 22- month impact scaled up to 24-months linearly Year 3 impact = average of Year 1 and Year 2 impacts
Health impact trajectory	Year 2 and Year 3 impacts = Year 1 impact	Year 2 impact = +50% Year 1 impact Year 3 impact = +50% Year 2 impact	Year 2 impact = -50% Year 1 impact Year 3 impact = -50% Year 2 impact
Proportion of healthcare costs due to earnings impact	0%	0%	25%
Proportion of travel costs due to earnings impact	25%	50%	10%
Proportion of childcare costs due to earnings impact	25%	50%	10%

Source: Economic evaluation spreadsheet

Appendix B – Expected effects of the IPS services

Change in output

Assuming that workers are compensated in line with their productivity, an increase in earnings (an outcome that was tested in the HLT impact assessment) would reflect an increase in output. Increased output is a positive economic benefit as this produces greater economic value for society.

Translating an increase in wages to an increase in output relies on the assumption that the increase in earnings is due to an increase in hours and/or productivity. However, an earnings increase may instead be due to a redistribution of output from the employer to the employee. This would not constitute an economic benefit as there would be no gains to society. Additionally, if employers took a proportion of any increase in output as profit, the rise in output might be greater than that which would be estimated when considering any increase in wages resulting from receipt of IPS.

Any direct rise in output as a result of receiving IPS services might cause additional indirect increases in output through multiplier effects. The economic multiplier refers to the additional impact on output of the increased expenditure of those in the treatment group who experience an increase in earnings due to receipt of IPS. As the IPS services were largely targeted at the unemployed, a large proportion of any increase in earnings was likely to be spent on increased consumption, rather than savings. There might also be multiplier effects from any increase in the earnings of the trials’ employment specialists, although it is more ambiguous whether any increase in earnings for this group would be consumed or saved. The economic evaluation follows the recommendations of the DWP Social Cost Benefit Analysis Framework to ignore multiplier effects, given the difficulty in measuring their magnitude (Fujiwara 2010).

An increase in employment for one individual in a multi-person household may result in changes in the household division of labour. If a recruit in the treatment group moved into employment or increased their earnings, others in the household might reduce their hours. This might be necessary to meet childcare requirements. Whilst there may nevertheless be a net benefit to the household, the net change in output resulting from receiving IPS might be lower than the aggregate increase in output resulting from the increase in earnings arising from the treatment group moving into employment.

If the IPS services did have an impact on working hours or pay for the treatment group, any substitution of labour within the household might have implications for the other expected financial and economic benefits. In this case, information on household composition collected at the time of randomisation could be used to explore the potential implications of any substitution effects for the BCRs.

Change in UC and legacy benefit payments

An increase in earnings would result in a decrease in UC payments where a recruit in the treatment group earned more than their UC work allowance and was therefore subject to the UC taper rate, currently set at 55%.^{[footnote 12]} This implies that for every £1 of additional earnings for a member of the treatment group, UC payments reduce by £0.45. An increase in earnings and/or movement into employment would also result in a decrease in legacy benefit payments, such as Employment Support Allowance (ESA), Job Seeker’s Allowance (JSA) and Income Support. A decrease in UC and legacy benefit payments is a financial benefit as this represents a net saving for the exchequer. It does not constitute an economic benefit as the savings to the exchequer are matched by a loss to those previously receiving these welfare benefits.

Change in other benefit payments and tax credits

An increase in earnings would reduce eligibility for other means-tested benefits, tax credits and other forms of government support. This would include Council Tax Support, Housing Benefit, Child Tax Credit, Working Tax Credit, Free School Meals, and Carer’s Allowance. Higher earnings would also affect student loan repayments. Any reduction in other benefits payments, tax credits and other forms of government support resulting from the IPS services would be a financial benefit as this represents a net saving to the exchequer. However, there would be no economic benefit as the savings to the exchequer would be matched by a loss to those previously receiving these welfare benefits.

An increase in the income of a recruit to the treatment group will, all other things being equal,^{[footnote 13]} result in an increase in household income. Certain benefits are means tested at the household level, for instance Housing Benefit. In order to consider the full effect of an increase in earnings on the receipt of other benefits and tax credits, information on household income would be desirable. However, as noted previously, information on household characteristics at the time of randomisation can be used to inform the assumptions about how any change in earnings might affect the receipt of other benefits and tax credits.

Change in operational costs

Operational costs refer to the marginal costs incurred by DWP related to the management of welfare benefits payments, for instance through the Counter Fraud and Compliance Directorate and the Operational Excellence Directorate. A reduction in UC and legacy benefit payments would result in a decrease in operational costs in addition to the saving from reduced benefit payments. This constitutes a financial benefit as it represents a net saving for the exchequer. It also results in an economic benefit as the resources saved from reduced operational costs can be reallocated to alternative uses.

Change in income tax revenue

An increase in earnings would increase the amount of income tax paid by recruits earning above their personal allowance. This would constitute a financial benefit only, as whilst tax revenues increase, there would be a corresponding loss of income for recruits paying the additional income tax.

Taxation can affect the behaviour of individuals and reduce economic efficiency. The potential impact on the cost-benefit analysis of all types of taxation considered in this report is discussed in section 2.4.6.

Change in employee National Insurance Contributions (NICs)

An increase in earnings results in an increase in NICs for trial participants earning above the primary threshold. This constitutes a financial benefit as the exchequer’s tax revenues increase. Increased revenues for the exchequer are offset by lost income for participants paying additional NICs and so there is no economic benefit. Change in employer NICs

An increase in earnings would increase NICs for employers provided the employee was earning above the secondary threshold. Again, whilst this would be a financial benefit as tax revenues increase, this would be offset by a reduction in revenue for the employer and so would not result in any economic benefit.

Change in revenue from indirect taxes

An increase in earnings is assumed to imply an increase in the consumption and production of goods which, when these are subject to indirect taxes such as VAT, will result in increased tax revenue. Once again, the financial benefit is not matched by an economic benefit due to the loss of income or revenue for those individuals or firms paying the additional indirect taxes.

Change in healthcare costs

An increase in employment is assumed to result in a reduction in the demand for NHS services and thus a reduction in the total cost of healthcare provision. This is because there is evidence that employment has latent health benefits, for example through the structured use of time, increased activity and social contact, the sense of collective endeavour, and socio-economic status (Jahoda 1982). Working can also produce health benefits due to the increase in income generally associated with gaining employment.

Fujiwara (2010) provides evidence of a causal relationship between employment status and NHS usage. He demonstrates that the reduction in healthcare costs is greater when an individual with worse health moves into work. A reduction in healthcare costs can be considered a financial benefit as it produces a net saving for the exchequer. In addition, savings in the cost of providing NHS services can be reallocated to alternative productive uses, resulting in an economic benefit to society.

Evidence suggests that the quality of work obtained is a factor in determining the impact of employment on health (Fujiwara 2010). There is mixed evidence from the HLT on the quality of employment obtained as a result of receiving the IPS services (points on which are contained in the synthesis report) so the analysis does not take into account how the quality of work affects healthcare costs.

Additionally, the trial population has higher morbidity and multi-morbidity rates than the general population. As a result, they would be expected to incur greater healthcare costs for the exchequer than the average individual. The intention was to use NHS Digital Data to estimate the impact of the IPS service on the use of health services. However, unfortunately it was not possible to gain access to NHS Digital data. The section on HRQOL explains the alternative approach used to monetise healthcare savings.

Change in travel costs

An increase in employment and earnings might be accompanied by an increase in travel to work by the recruit and negative externalities such as greenhouse gas emissions. Increased travel is expected to have an economic impact through the potential for a negative impact on health, wellbeing and other factors. Any financial implications for the exchequer, for example due to any increase in the use of health services by those experiencing respiratory problems arising from increased pollution, are difficult to estimate or monetise and so are not considered in the analysis presented here.

Change in childcare costs

An increase in employment and earnings is assumed to result in an increase in childcare costs as parents have less non-labour time to care for their children. The lost income for recruits paying additional childcare costs has implications for the economic benefits of the IPS services. There is also the potential for indirect effects on revenue for childcare providers, affecting the economic case. There is no direct financial benefit. Although tax revenues from childcare providers may increase Fujiwara (2010) does not consider any indirect financial benefits, given the lack of knowledge of the market structure for childcare. This report follows the same approach of focusing on the economic implications of any increase in the use of childcare, based on estimates of the impact of the IPS services on employment.

Change in health-related quality-of-life

As well as its focus on achieving employment-related outcomes, IPS is expected to result in improvements in health-related quality of life (HRQOL). This constitutes an economic benefit as it represents a monetisable gain to participants and society. However, it is less certain whether the improvement in HRQOL would reduce use of healthcare services and the costs to the exchequer of healthcare provision. In common with other studies, it is assumed that any impact from the IPS service on HRQOL would result in economic, rather than financial, benefits. Nevertheless, if changes in HRQOL increased the likelihood of recruits entering work or being more productive in work, they might have an indirect impact on the exchequer. This includes the saving resulting from reduced use of health services by those in employment, noted earlier.

Over a longer timeframe, an improvement in health as a result of support from the IPS services is expected to improve life expectancy. However, as this analysis considers benefits accruing over a period of 3 years following randomisation, this potential benefit cannot be included.

Appendix C - Characteristics of recruits in the treatment group

Table C. 1 Number of recruits in the treatment group by cohort and trial group

	Cohort 18 to 19	Cohort 19 to 20	Total
SCR IW	580	680	1,260
SCR OOW	831	968	1,799
WMCA	862	975	1,837
All OOW	1693	1943	3,636

Source: Economic evaluation spreadsheet

Table C. 2 Proportion of non-parents, lone parents and couples with children by trial group

	Not a parent	Lone parent	Couple with children
SCR IW	73.9%	9.8%	16.3%
SCR OOW	76.7%	13.0%	10.3%
WMCA	76.4%	12.7%	10.9%
All OOW	76.5%	12.8%	10.6%

Source: Economic evaluation spreadsheet

Table C. 3 Proportion of recruits to the treatment group that are ESA claimants by year, by trial group1 %

	Year ESA claimants (Treatment)	Year 2 % ESA claimants (Treatment)	Year 3 % ESA claimants (Treatment)
SCR IW	4.4%	4.6%	4.6%
SCR OOW	12.7%	8.1%	5.7%
WMCA	8.6%	5.5%	4.0%
All OOW	10.6%	6.8%	4.8%

Source: Economic evaluation spreadsheet

Table C. 4 Average proportion of a full-time job worked by each trial group

	Weeks worked per month employed	Proportion of full-time job
SCR IW	4.1	0.94
SCR OOW	3.5	0.80
WMCA	3.5	0.80
All OOW	3.5	0.80

Source: Economic evaluation spreadsheet

Appendix D – Breakdown of total and annual economic benefits by sources under the optimistic and pessimistic scenarios

Figure D. 1 Total and annual economic benefits from each source for the optimistic scenario, SCR IW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reduced healthcare costs	£10,191	£18,669	£16,418	£9,875	£55,153
Increased childcare costs	-£1,788	-£3,273	-£2,876	-£1,730	-£9,667
Increased travel costs	-£4,159	-£7,615	-£6,691	-£4,024	-£22,489
Social cost (or benefit) of Exchequer Finance	£332,828	£115,876	£62,920	-£1,975	£509,649
Improved health-related quality of life	£822,301	£2,505,152	£4,278,308	£2,755,211	£10,360,972

Source: Economic evaluation spreadsheet

Figure D. 2 Total and annual economic benefits from each source for the pessimistic scenario, SCR IW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Reduced healthcare costs	£7,643	£13,867	£8,803	£3,331	£55,153
Increased childcare costs	-£3,218	-£5,835	-£3,701	-£1,400	-£9,667
Increased travel costs	-£7,486	-£13,575	-£8,610	-£3,257	-£22,489
Social cost (or benefit) of Exchequer Finance	£612	£1,109	£702	£265	£0
Improved health-related quality of life	£822,301	£1,421,174	£791,283	£267,978	£10,360,972

Source: Economic evaluation spreadsheet

Figure D. 3 Total and annual economic benefits from each source for the optimistic scenario, SCR OOW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Social cost (or benefit) of Exchequer Finance	£534,056	£201,450	£111,609	£0	£847,114
Improved health-related quality of life	£935,113	£3,451,549	£6,548,516	£4,201,924	£15,137,102

Source: Economic evaluation spreadsheet

Figure D. 4 Total and annual economic benefits from each source for the pessimistic scenario, SCR OOW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Increased childcare costs	-£1,814	-£4,600	-£6,247	-£3,829	-£16,490
Increased travel costs	-£4,613	-£11,697	-£15,888	-£9,738	-£41,936
Reduced healthcare costs	£11,751	£29,329	£39,036	£23,763	£103,879
Social cost (or benefit) of Exchequer Finance	£759,327	£426,544	£158,227	-£4,752	£1,339,347

Source: Economic evaluation spreadsheet

Figure D. 6 Total and annual economic benefits from each source for the pessimistic scenario, WMCA

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Substitution effects	£551	£1,453	£1,665	£813	£4,483
Increased childcare costs	-£3,265	-£8,200	-£8,899	-£4,248	-£24,612
Increased travel costs	-£8,304	-£20,853	-£22,632	-£10,803	-£62,591
Reduced healthcare costs	£8,813	£21,785	£23,206	£10,984	£64,790

Source: Economic evaluation spreadsheet

Figure D. 7 Total and annual economic benefits from each source for the optimistic scenario, all OOW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Social cost (or benefit) of Exchequer Finance	£1,295,733	£633,860	£277,643	£0	£2,207,236
Improved health-related quality of life	£1,635,935	£5,496,139	£9,961,795	£6,338,635	£23,432,504

Source: Economic evaluation spreadsheet

Figure D. 8 Total and annual economic benefits from each source for the pessimistic scenario, all OOW

	2018 to 2019	2019 to 2020	2020 to 2021	2021 to 2022	Total
Improved health-related quality of life	£1,635,935	£2,964,910	£1,832,160	£616,509	£7,049,515

Source: Economic evaluation spreadsheet

Appendix E – Components of the economic case under the optimistic and pessimistic scenarios

Figure E. 1 Components of the economic case for SCR IW trial group, optimistic scenario

Economic case	Amount
Reduced healthcare costs	£55,153
Increased childcare costs	-£9,667
Increased travel costs	-£22,489
Social cost (or benefit) of Exchequer Finance	£509,649
Improved health-related quality of life	£10,360,972
Cost	£2,603,396
Net benefit	£8,290,222

Source: Economic evaluation spreadsheet

Figure E. 2 Components of the economic case for SCR IW trial group, pessimistic scenario

Economic case	Amount
Reduced healthcare costs	£33,644
Increased childcare costs	-£14,154
Increased travel costs	-£32,928
Substitution effects	£2,688
Improved health-related quality of life	£3,302,735
Cost	£2,603,396
Net benefit	£688,589

Source: Economic evaluation spreadsheet

Figure E. 3 Components of the economic case for all OOW trial group, optimistic scenario

Economic case	Amount
Social cost (or benefit) of Exchequer Finance	£2,207,236
Improved health-related quality of life	£23,432,504
Cost	£11,036,181.46
Net benefit	£14,603,559

Source: Economic evaluation spreadsheet

Figure E. 4 Components of the economic case for all OOW trial group, pessimistic scenario

Economic case	Amount
Improved health-related quality of life	£7,049,515
Cost	£11,036,181.46
Net benefit	£3,986,667

Source: Economic evaluation spreadsheet

Figure E. 5 Components of the economic case for SCR OOW trial group, optimistic scenario

Economic case	Amount
Social cost (or benefit) of Exchequer Finance	£847,114
Improved health-related quality of life	£15,137,102
Cost	£4,235,571
Net benefit	£11,748,644

Source: Economic evaluation spreadsheet

Figure E. 6 Components of the economic case for SCR OOW trial group, pessimistic scenario

Economic case	Amount
Improved health-related quality of life	£4,357,026
Cost	£4,235,571
Net benefit	£121,454

Source: Economic evaluation spreadsheet

Figure E. 7 Components of the economic case for WMCA trial group, optimistic scenario

Economic case	Amount
Increased childcare costs	-£16,490
Increased travel costs	-£41,936
Reduced healthcare costs	£103,879
Social cost (or benefit) of Exchequer Finance	£1,339,347
Cost	£6,800,610
Net benefit	£5,415,810

Source: Economic evaluation spreadsheet

Economic case	Amount
Increased childcare costs	-£24,612
Increased travel costs	-£62,591
Reduced healthcare costs	£64,790
Cost	£6,800,610
Net benefit	£6,818,541

Source: Economic evaluation spreadsheet

1. Note that as the economic evaluation only considers a 3 year period after randomisation, the choice of discount rates used will have little impact on the results. This would be more of an issue when evaluating a programme with benefits accruing over decades, such as major infrastructure investments. ↩

2. In SCR a concessionary fare scheme was available to those participating in an employment support project for whom travel was likely to be a significant barrier to work. It is not known how long concessionary fares were available after entering work, but anecdotally, uptake was reported to be low. ↩

3. Devlin et al. (2018) provides a detailed description of how EQ5D can be converted to a measure of HRQOL. ↩

4. This is estimated using an area under the curve calculation. ↩

5. Different numbers of recruits started on the trial in the 2018 to 2019 and 2019 to 2020 financial years and the benefits for each cohort were estimated over a period of 3 years. ↩

6. These were categorised as follows: time spent training on trial-specific tasks, time spent training on intervention-specific tasks, raising awareness of the service (for example with GPs, NHS trusts, community health services and IAPT services), processing referrals and arranging the initial appointment, conducting the randomisation interview, liaison with employers, and other tasks. ↩

7. Some examples of site delivery-specific set-up costs may include the costs of publicity to generate referrals and setting up customer relations management systems. These would have been required even if there was no intention to evaluate the impact of the IPS service. ↩

8. This includes the following: processing referrals and arranging the initial interview, conducting the randomisation interview, and other tasks. Because these activities were delivered to both treatment and control groups the proportion of time spent on each was halved. The proportion of staff time spent on receiving training on trial-specific tasks is included in full when estimating trial-specific costs. ↩

9. One of the providers in WMCA did not provide a breakdown of costs by trial period; this has been estimated using proportions calculated from the other two providers. ↩

10. In a small number of cases there was uncertainty about the hours worked by past members of staff and so it was assumed that they worked 0.8 of the hours of a full-time equivalent. ↩

11. Measuring Administrative Costs: UK Standard Cost Model Manual 2005, Better Regulation Executive, Cabinet Office. Overheads include a range of costs relating to administrative activities: payroll overheads such as employer pension contributions and NI contributions; human resource costs such as training and development; IT costs such as hardware, software and telephone and communications; and personal and general indirect costs such as office articles and subscriptions, and insurance and rent costs, respectively. Thus, National Insurance and pension costs are removed from the total cost calculations after which the 30% rate is applied. ↩

12. Universal Credit and earnings - GOV.UK (www.gov.uk) ↩

13. This assumes that there are no substitution effects within the household. ↩