Research and analysis

Health-led Trials: Evaluation Synthesis Report

Published 20 April 2023

12-month outcomes evidence synthesis

August 2022

DWP research report no.1025

A report of research carried out by the Institute for Employment Studies on behalf of the Department for Work and Pensions.

Crown copyright 2023.

You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence. To view this licence, visit The National Archives or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email [email protected]

This document/publication is also available on GOV.UK.

If you would like to know more about DWP research, email [email protected]

First published April 2023.

ISBN 978-1-78659-516-4

Views expressed in this report are not necessarily those of the Department for Work and Pensions or any other government department.

Executive summary

This report presents findings from the evaluation of the Health-led Employment Trials (HLTs). These tested the provision of Individual Placement and Support (IPS) – a well-evidenced voluntary employment support programme for people living with severe and enduring mental illness in secondary care – with a group experiencing mild/moderate mental and/or physical health conditions in primary and community care settings. Estimates of impact on employment, earnings, health and wellbeing used a randomised controlled trial (RCT) design, with outcomes taken from linked survey and administrative data sets. An economic evaluation estimated the value of the impacts set against the costs of delivering the IPS services. A process evaluation using surveys and qualitative research explored implementation.

The trials recruited 9,785 people across 2 sites between May 2018 and October 2019. In Sheffield City Region (SCR), 6,110 people were recruited including an out-of-work group (SCR OOW) and a group in employment but struggling (SCR IW). The West Midlands Combined Authority (WMCA) trial recruited 3,675 people, all of whom were OOW. The IPS service offered support for a total of 12 months, with 9 months of support to find employment and 4 further months of in-work support. The evaluation found:

• Trial participants had many barriers to work: Many recruits had not worked for 2 years, and some had never worked. It was common for recruits to have 6 or more interacting health conditions.
• Improving health condition management and achieving health referrals alongside employment support, built people’s capability and self-belief. The treatment group appreciated being able to focus on job roles that were matched to their goals and capabilities.
• Impacts on employment, health and wellbeing varied by site and trial group:
  • In WMCA, where all recruits were OOW, there was a substantial and strongly significant impact on the probability of being employed for 13+ weeks over the year following randomisation.
  • The SCR IW group saw less substantial and weaker impact on employment using the same measure. No impact on employment was observed for the SCR OOW group.
  • Across SCR groups, strongly significant impacts were seen for health and wellbeing outcomes. These did not emerge in WMCA.
• Health outcomes produced a stronger return to society and the exchequer than employment outcomes. This led to a return-on-investment for every £1 invested in the IPS services of £0.01 in WMCA, and in SCR, of £2.02 (SCR OOW) and £2.32 (SCR IW) and £1.22 for the pooled out of work group.
• Progress amongst the treatment group on key “movement to work” measures: Trial designers anticipated that impacts would be preceded by improvements to: jobsearch capability, use of health services, and self-confidence. The evaluation showed progress amongst the treatment group on all these dimensions.
• High levels of satisfaction with IPS support, for example
  • 68% of participants said that their employment specialist “understands my needs a lot”.
  • 69% of participants reported that their employment specialist “has the right skills and expertise”.
• Different leadership structures and different delivery models were the factors most likely to explain the differences in impact between the sites.
  • specialised employment advisers (IW or OOW but not both) and smaller caseloads enabled more employer engagement activity.
  • A mix of short and longer meetings was likely to provide momentum for employment outcomes. Less frequent, longer meetings were likely to enable focus on making best use of health support, as well as wellbeing.
  • At a strategic level, building integration between health and employment systems, and increasing understanding of the value of work to health and wellbeing outcomes are important factors for future delivery.

Acknowledgements

The evaluation of the Health-led Employment Trials was commissioned by the joint Work and Health Unit (WHU) which is sponsored by the Department for Work and Pensions and the Department of Health and Social Care. Throughout the design of the trials and evaluation methodology several colleagues within this directorate have made significant contributions. The authors are especially grateful to David Johnson, Huw Meredith, Chris Gunning, Anna Silk, Laura Mitzel, Tracey Gent, Nisha Patel and Karen Ruby. We are indebted to Hayley Moore-Purvis who had a critical role in the design and set-up of the information governance of the trials, ensuring its data management practices were effective, and robust.

The Office for National Statistics was the data safe haven for the evaluation, and the authors would like to thank the team involved for their work, especially Lynne Liddell.

The authors would like to thank colleagues within the Sheffield City Region (SCR) for their support with the evaluation, for taking the time to host and participate in workshops, in particular Andrea Fitzgerald, Katy Pugh, and Adam Whitworth. Within South Yorkshire Housing Association (SYHA), Niall O’Reilly and Megan Staley were hugely supportive in enabling the evaluation consortium to interview employment specialists, health partners and employers. Finally, our thanks go to everyone in these roles who participated in an interview for the evaluation.

Within the West Midlands Combined Authority (WMCA), we would like to thank the central team for their support for the evaluation, especially Anita Hallbrook and Ethan Williams. Our thanks also go to staff at the delivery partners in the region, including Remploy, Dudley and Walsall Mental Health Trust, and Prospects for their support and engagement in the evaluation activities.

We would also like to thank staff at NHS England, who part-funded the trials, and particularly to Elizabeth Wade, Elizabeth Dyer, Shabana Janjua, Kevin McKenna, Belinda McIntosh and Johanna Ejbye, for their contributions to trial development and delivery.

Our special thanks and gratitude go to all the trial recruits who took the time to participate in the quantitative and qualitative evaluation research. Some took part at several points over the course of a year, and all shared their views and experiences about their health and work so candidly. Without their participation the evaluation would not have been possible.

Lastly, the authors would like to thank colleagues within the wider research teams in the evaluation consortium. Stephen Bevan is the Chief Investigator has provided invaluable subject matter expertise to the evaluation. Emma Disley from RAND Europe made a substantial contribution in guiding the ethics processes and supporting the trials through Health Research Authority (HRA) approval. Within the Institute for Employment Studies (IES) we are especially grateful to Sara Butcher and Jade Talbot for their administrative support, and Rebecca Duffy for proofreading and formatting the reports. From the research team we are sincerely grateful to Jonathan Buzzeo, Ceri Williams, Georgie Akehurst, Kate Alexander, Rachel Cetera, Sally Wilson, Zofia Bajorak, Billy Campbell, and Rahul Shakala for their help with interviews and analysis. We are indebted to the team at Qa Research, and particularly Helen Hardcastle and Patrick Gower who supported us to recruit non-trial employers to understand their recruitment practices in the pandemic.

Any errors or omissions in this report are the responsibilities of the authors.

Authors’ credits

Becci Newton is Director of Public Policy and Research at the Institute for Employment Studies (IES) and specialises in research on unemployment, inactivity, health, skills and labour market transitions. Becci has managed the evaluation since its design and contributed to the process evaluation. She has led multiple evaluations for DWP including of the 2015 ESA Reform Trials and the Work Programme.

Rosie Gloster is a Principal Research Fellow at IES. She supported the management of the evaluation consortium and contributed to the process evaluation. She is a mixed-methods researcher specialising in employment, and careers. She has authored several reports for the Department for Work and Pensions (DWP), including the Evaluation of Fit for Work.

Joanna Hofman is a Research Leader at RAND Europe focusing on research and evaluation in the fields of employment and social policy. She has led several evaluations of the Individual Placement and Support (IPS) including of IPS for Alcohol and Drug Dependence for Public Health England (PHE). She contributed to evidence reviews that informed the trial design and this synthesis report and advised throughout the evaluation.

This report draws on all evaluation strands that were produced and authored by:

Richard Dorsett is Professor of Economic Evaluation at the University of Westminster. He has worked on numerous impact evaluations, mostly in the fields of employment, welfare and education/training. He led on the statistical design of the trials and the impact analysis.

James Cockett is a Research Fellow (Economist) at the Institute for Employment Studies. He has supported the 4-month impact analysis of the Health-led Trials. He is a Labour Economist with a particular interest in labour market transitions of disadvantaged groups. James has authored reports for both the Low Pay Commission and the Social Mobility Commission in his early career.

Charlotte Edney is a Research Fellow at IES. She completed her PhD in Economics at Lancaster University where she gained experience in using a wide range of quantitative methods. In her work she has used various UK household surveys, cohort studies and administrative data.

Helen Gray was a Principal Research Economist at IES. She has particular expertise in the causal identification of impact using quantitative methods and linked administrative data sets. As well as leading the economic evaluation of the HLTs, she was responsible for the manipulation of DWP and HMRC data for the impact evaluation.

Dan Muir is a Research Officer at IES. He has supported the evaluation’s 12-month impact analysis as well as the economic evaluation. Dan joined IES in September 2021 after completing his MSc Economics studies at the University of Bristol. He has experience working with a range of large data sets and quantitative analysis in various policy-related projects.

Matthew Gould is a Lecturer in Economics at Brunel University London. His research specialises in applying theory and computation to a range of economic problems. He has worked on several projects focused on education, employment and taxation. He led on the development and maintenance of the randomisation tool.

Joe Crowley is a Senior Researcher in the Health and Social Care team at NatCen Social Research, whose main methodological expertise lies in survey data collection and analysis. He has worked across a variety of large-scale cross-sectional surveys, including NatCen’s flagship British Social Attitudes survey and the Family Resources Survey. He has also worked on a range of mixed-methods projects, combining qualitative and quantitative methods.

Imogen Martin is a Researcher in the Health and Social Care team at NatCen Social Research. She has worked on areas across the policy spectrum including healthcare, disabilities and welfare. As part of the Health-led Employment Trials, Imogen has assisted with the delivery and analysis of the interim and final surveys.

Phoebe Weston-Stanley is a Research Assistant in the Health and Social Care team at NatCen Social Research. She has worked on projects using both qualitative and quantitative methodologies across the policy spectrum. As part of the Health-led Employment Trials, Phoebe has assisted with the analysis of the final surveys.

Dr Rosa Lau is a Research Director in the Health and Social Care team at NatCen Social Research. She has extensive experience of delivering complex applied research and service evaluations using mixed methods on areas including healthcare, public health, employment, housing and welfare.

Dr Priya Khambhaita is Co-Director of Health and Social Care at NatCen Social Research. She has 12 years of experience in conducting policy and academic research on health, social care, work, and income.

Jess Elmore is a Researcher at Learning and Work Institute. She contributed to the process evaluation. She is a qualitative researcher who has worked across a range of projects with a focus on disadvantaged or hard-to-reach groups and their access to education and employment.

Rebecca Duffy, IES Project Support Officer, proofread and formatted the report.

Glossary of terms

Area	A defined, geographic area within a trial site
Base	The number of observations or cases in a sample. For example, a survey may have a base=2,300 respondents. During analysis the base may become smaller, for example if not all respondents answer a particular question, or when analysing responses from a subset of the full sample
Baseline data collection	Data from the baseline assessment completed by provider staff who recruited people to the trial
Binary variable	A variable measured with only 2 possible values, for example hot and cold, 0 and 1, or happy and unhappy. More complex variables (such as a happiness scale from 1-7) are sometimes re-coded as binary values during analysis
Bivariate analysis	The analysis of 2 variables for the purpose of determining the statistical relationship between them
Causal link	The connection between a cause and an effect
Clinical Commissioning Groups	Clinically-led statutory NHS bodies responsible for the planning and commissioning of healthcare services for their local area
Controlling for	In statistical modelling with multiple variables and factors, keeping 1 variable constant in order to examine and test the relationship and effect between other variables of interest in the model
Correlation	In statistics, the association or relationship between 2 variables, not necessarily causal. For example, the rings in a tree trunk increasing with the age of the tree is an example of positive correlation
Data set	A collection of data or information such as all the responses to a survey or all the recordings from a set of research interviews
Deep dive	Thematic case studies used in the process and theory of change evaluation with methods varying depending on the selected themes for investigation
Demographic	A particular section of the population. Also refers to characteristics of an individual of interest for research, such as age, gender, and ethnicity
Derived variable	A variable that was not directly asked in a survey, but created at analysis stage, for example by merging 2 or more variables
Descriptive analysis	Producing statistics that summarise and describe features of a data set such as the mean, range and distribution of values for variables
EuroQol-5D-5L (EQ5D5L)	Descriptive system for health-related quality of life states in adults, consisting of 5 dimensions (Mobility, Self-care, Usual activities, Pain & discomfort, Anxiety & depression), each of which has 5 severity levels described by statements appropriate to that dimension
Employment specialists	Staff employed by the trials to undertake randomisation appointments, provide IPS support to the treatment group, and undertake employer engagement
Final survey	The survey completed by participants 12 months after randomisation
Job search self-efficacy	9 item scale to measure self-efficacy relating to finding employment
4-month survey	The survey completed by trial recruits 4 months after starting the trial
Intervention	The work and health support provided in Sheffield City Region and the West Midlands Combined Authority as part of the trial
In employment/working	Those in employment full-time, part-time, or less than 16 hours a week; those who are self-employed
In paid work	Those in those in employment full-time, part-time, or less than 16 hours a week, not those who are self-employed
Individual Placement and Support (IPS)	IPS is a voluntary employment programme that is well evidenced for supporting people with severe and enduring mental health needs in secondary care settings to find paid employment
IPS fidelity scale	A scale developed to measure the degree to which IPS interventions follow IPS principles and implement evidence-based practice
Longitudinal surveys	Repeated surveys that study the same people over time
Multi-morbidity	The occurrence of multiple chronic conditions within the same individual with no single condition holds priority over any of the co-occurring conditions. This term has been selected as the evaluation consortium does not hold information about the main condition affecting individuals
Participants	Trial recruits allocated to treatment, who went on to receive support, as indicated by having 1+ meetings with an employment specialist. This is used to differentiate between those who experienced limited support beyond randomisation as opposed to those whose support was more extensive. Other terms are used to describe people taking part in the trial (recruits) and people taking part in the surveys (respondents) – see below
Prevalence	The extent to which something occurs in a population or group, often expressed as a percentage
Provider staff	Those working in provider organisations including employment specialists delivering IPS support, as well as managers and administrators
p-value	Used as a measure of statistical significance. Low p-values indicate results are very unlikely to have occurred by random chance. p<0.05 is a commonly cited value, indicating a less than 5% chance that results obtained were by chance. Research findings can be accepted with greater confidence when even lower p-values are cited, for example p<0.01 or p<0.001
Randomised controlled trial	A study to test the efficacy of a new intervention, in which participants are randomly assigned to 2 groups: the intervention group receives the treatment, while the control group receives either nothing or the standard current treatment
Recruits	People who agreed to take part in the trials and who were randomised to either the treatment or control group
Refer / referral	A recommendation that an individual should be considered for the trial, facilitated by a means to directly connect them to a trial provider
Respondents	Trial recruits from the treatment or control group who were invited to take part in the evaluation and took part in the surveys. As such the descriptive analysis of the survey identifies treatment group respondents and control group respondents
Self-refer / self-referral	Individual applies for more information about the trial via the trial website or helpline and uses information there (phone number, web form, email) to make contact with the trial provider and request support
Signpost	Recommendation to an individual from a support organisation that they consider joining the trial, by providing them with information (leaflets, reference to website or helpline) leading potentially to the individual self-referring into the trial
Site	The trials were delivered in 2 combined authorities, which are termed sites
Statistical significance	Statistical significance indicates that the result or difference obtained following analysis is unlikely to be obtained by chance (to a specified degree of confidence) and that the finding can be accepted as valid. A study’s defined significance level is the probability of the study rejecting the null hypothesis (that there is no relationship between 2 variables), demonstrated by the p-value of the result
Short Warwick-Edinburgh Mental Well-being Scale	The SWEMWBS is a short version of the Warwick–Edinburgh Mental Wellbeing Scale (WEMWBS). The WEMWBS was developed to enable the monitoring of mental wellbeing in the general population and the evaluation of projects, programmes and policies which aim to improve mental wellbeing
Survey	A research instrument used to collect data by asking scripted questions or using lists or other items to prompt responses. Can be conducted in person face-to-face, by telephone, or by postal or web-based questionnaire
Sustainability and Transformation Partnership	A partnership of local NHS organisations and Councils which develops proposals for improved healthcare
Tenure	Housing arrangement or status of an individual, for example owner-occupier, private renter, or local authority or housing association renter
Theory of Change (ToC)	A description and illustration of how and why a desired change is expected to happen in a particular context. It sets out the planned major and intermediate outcomes and how these relate to each other causally
Thrive into Work	The name given to the trial in WMCA
Trial arm	This is used to denote the allocation of individuals to either the treatment or control group, with these groups known as the trial arms
Trial group(s)	3 trial groups are referred to in the report: 2 out-of-work (OOW) groups (one in each combined authority), and an in-work (IW) group in Sheffield City Region (SCR). These groups are pooled as All OOW and All SCR in different elements of the analysis
Variable	A variable is defined as any individual or thing that can be measured
Weighting	During analysis of survey data, adjusting for over- or under-representation of particular groups, to ensure that the results are representative of the wider population
Working Win	The name given to the trial in SCR

1. Summary

1.1. Rationale for Health-led Employment Trials

In their landmark report Waddell and Burton (2006) posed the question: “Is work good for your health and wellbeing?”. The evidence reviewed clearly highlighted a link: that “work is generally good for physical and mental health and wellbeing…work can be therapeutic and can reverse the adverse health effects of unemployment”. The review also found that the quality of work is important: “The provisos are that account must be taken for the nature and quality of the work and its social context; jobs should be safe and accommodating.”

More recently, Marmot et al. (2020) provided compelling evidence of the links between quality of work and health and wellbeing. This review showed that employees in lower-status work had poorer health and lower life expectancy than those in higher-status roles, and experienced more stressors, which had health implications. Marmot’s concept of the so-called “social gradient” in health applies in organisations as well as in wider society.

People with long-term health conditions and disabilities have lower rates of labour market participation than those without. In January to March 2015, just before the health-led employment trials (HLTs) received funding for development, the difference between employment rates of non-disabled (79%) and disabled people (46.3%) – known as the disability employment gap - was 32.7 percentage points (ppts). In January to March 2022 there were an estimated 9 million working-age disabled people, of which 4.8 million were in employment (53.8%). The equivalent employment rate for non-disabled people was 82%. This means that, while the disability employment gap has decreased over the last 7 years it still stands at 28.2 ppts (ONS, 17 May 2022).

The origins of the HLTs lie in 2015, when the Work and Health Unit (WHU) – a joint unit between the Department for Health and Social Care (DHSC) and Department for Work and Pensions (DWP) working with NHS England – was established and secured funding to develop, deliver and test new ways of working across health and work to improve individual economic, social and clinical outcomes within the objective of increasing employment amongst disabled people. This funding supported the design and implementation of the HLTs which tested Individual Placement and Support (IPS) as the means to integrate employment and health support.

1.2. About IPS

IPS is a well-evidenced voluntary employment programme for supporting people with severe mental health needs in secondary care settings to find paid employment (see for example, Wallstroem et al. (2021), Bond et al. (2020), Frederick & VanderWeele (2019), Metcalfe et al. (2018), Modini et al. (2016). It is based on 8 principles:

• It aims to get people into competitive employment
• It is open to all those who want to work
• It tries to find jobs consistent with people’s preferences
• It works quickly
• It brings employment specialists into clinical teams
• Employment specialists develop relationships with employers based upon a person’s work preferences
• It provides time unlimited, individualised support for the person and their employer
• Access to specialist benefits counselling is included
  (IPS Employment Centre, undated)

IPS defines competitive employment as a job that any person can apply for regardless of disability status. Jobs may be full- or part-time and self-employment is included. They should offer at least minimum wage and those entering them should receive similar wages and benefits to their co‐workers.

The HLTs tested IPS-LITE which is a time-limited service (Burns et al. 2015). They investigated whether IPS-LITE was effective in primary and community healthcare settings for people experiencing self-defined low to moderate mental and physical health conditions. The trials tested support for people who were out of work (OOW) as well as people who were in work (IW) but struggling with their health conditions at the time of entering support. They were an early attempt to expand IPS to new populations (see also, Reme et al. (2015), Otomanelli et al. (2014), Coole et al. (2012), Li-Tsang et al. (2008), Magura et al. (2007)).

A Fidelity Scale is used to measure the degree to which IPS interventions implement evidence-based practice (Becker et al, 2019). Examples of the measures include a focus on the size of caseload with highest scores available where the average caseload is less than 15 per employment specialist (score of 5), and lowest (score of 1) where employment specialists have caseloads of 41 or more. The literature shows in practice the maximum caseload varies between 20 and 25 clients per employment specialist (Burns et al., 2015; Swanson et al., 2008 cited in: Fergusson et al., 2012; Burns & Catty, 2008). When the HLTs were commissioned, it was expected that the maximum caseload for a full-time employment specialist would be 25-30 in the first year of operation, but that this could rise to 30-35 as the service matured.

1.3. Trial design and eligibility

Two local sites – Sheffield City Region (SCR) and West Midlands Combined Authority (WMCA) – were selected in a competitive bidding process by the Work and Health Unit (WHU) to design and lead trials of new IPS-LITE services to improve health and work outcomes. Alongside, the evaluation consortium were commissioned to design and undertake a robust, national evaluation that could examine the effectiveness of the service, and detect overarching themes across the trial site, as well as local nuances.

Eligibility for the trials covered: being able to give informed agreement to take part and being aged over 18; and motivation to take part in voluntary employment support while not receiving any employment support beyond standard Jobcentre Plus services. Recruits with severe health conditions could not be included. Local eligibility in SCR covered individuals who were out of work (OOW) with a self-defined low/moderate mental health and/or physical health condition which was an obstacle to being in employment and individuals in employment (in work; IW) but who were either off sick or struggling in the workplace due to a self-defined low/moderate mental health and/or physical health condition. In WMCA the trial was open to people who had been OOW for more than 4 weeks, who wanted to find work, and who were disabled or had a health condition which presented an obstacle to them gaining work.

Figure 1.1 details the sequencing of the key time points during the design and implementation of the trial. The referral window for the trials was initially planned to be 12 months, but this was extended in November 2018 to cover an 18-month period (see section 1.7). The figure also illustrates the differences in the maturity of the regional organisations leading the trials, with SCR formed in 2013, and WMCA in mid-2016, and highlights the start of the COVID-19 pandemic in March 2020, that is, during the support period for the final cohorts entering the trial. The first national lockdown commenced on 23 March 2020.

Figure 1.1: Key time points during the design and implementation

1.4. Evaluation purpose and approach

The national evaluation aimed to answer the following questions:

• What impact, if any, does the provision of IPS services to the client groups have upon them attaining and sustaining employment and benefit receipt?
• What impact, if any, does the provision of IPS services to the client groups, have upon self-reported health, self-management of health and wider wellbeing, and upon health service usage?
• What costs are incurred and what benefits arise from the provision of IPS services to the client groups?
• How are any impacts upon sustained employment, benefits receipt, health and wellbeing achieved? What is the causal pathway to these impacts? How might poor or negative outcomes for some in the treatment group be explained? What system-level characteristics (for example, stakeholder cooperation, relationships with employers, awareness among GPs) need to be in place if similar interventions are to be successful and adopted in other locations or settings?

These questions form the focus for this synthesis report which covers all evaluation evidence. Additionally, the synthesis is an opportunity to consider:

• What lessons can be learned from trial delivery for: disabled people’s employment, for IPS delivery and for best implementation?

The evaluation methodology is summarised in Figure 1.2. Full details are supplied in Appendices – Chapter 6.

Figure 1.2: Overview of evaluation approach

1.5. Ethics and information governance

It was crucial to ensure the trials were conducted ethically, ensured the safety and wellbeing of recruits and researchers, and obtained the necessary permissions from research governance and ethics boards. Because the trials took place in the health system, it was necessary to apply for approval from the Health Research Authority (HRA). The evaluation consortium led on identifying potential ethical issues and proposed mitigation measures. It designed key documents and materials to be used in the trial which included: Participant Information Sheets (in accessible and plain English versions); trial agreement forms (in accessible and plain English versions); trial opt-out forms; consent and opt-out materials for the research (including surveys, interviews and observations). Given the importance of these materials, multiple stakeholders were engaged in development including from the trial sites, WHU, the consortium members, wider stakeholders and patient groups.

The trial agreement materials sought permission from recruits for their personal information to be collected during the trials and retained for up to 3 years after they ended, with personal information used for service delivery, to be shared and linked to other information held by government departments and NHS Digital for research purposes. Recruits also agreed that after 3 years, an anonymised version of their information could be stored in the UK Data Archive to support future research.

The materials made clear that recruits had the right to withdraw from the trials and/or the IPS service prior to the planned end. Where they requested to withdraw, the information they had already provided up to that point would continue to be used for research and analysis and could be linked to administrative data, but these recruits were not asked to participate in any further research, such as surveys or interviews.

The trials were considered by the HRA Research Ethics Committee (REC) on 19 September 2017 and HRA approval was issued on 1 November 2017. After this, the evaluation consortium requested the necessary local Capability and Capacity (C&C) checks in each of the 16 trusts and the CCGs across SCR and WMCA. The National Institute for Health Research (NIHR) requires HRA-approved research studies to provide information on the number of recruits via its Central Portfolio Management System (CPMS). These were provided monthly and enabled payments (according to agreements in the sites) to healthcare partners who referred people to the trial.

In accordance with best practice on transparency, the trials were registered on 28 October 2019. A Statistical Analysis Plan (SAP) setting out in advance how trial data would be analysed to understand impact was uploaded to the registration sites on 20 December 2019. The trials’ registrations can be found at the following links:

• Title: Sheffield City Region Health-led Trial Trial ID: ISRCTN68347173 Date registered: 28 October 2019

• Title: West Midlands Combined Authority Health-led Trial
  Trial ID: ISRCTN17267942 Date registered: 28 October 2019

WHU was the data controller for the trials and led the design of the data architecture and data flows. The Office for National Statistics (ONS) was appointed by WHU to be the data safe haven.

1.6. Randomisation preparation and delivery

Randomisation was carried out by employment specialists during an initial meeting, using a bespoke randomisation and data collection tool provided by the evaluation consortium which was accessed via a secure website. The tool took employment specialists through the process of screening individuals for eligibility; requested and recorded agreement to take part in the trial; collected personal data, national identifiers, background information and pre-trial responses to questions covering health and wellbeing and job search self-efficacy using a baseline data collection survey. The final stage involved the tool randomising recruits either to the IPS service (the treatment group) or the control group. The randomisation used a pre-specified algorithm, specifically a permuted-block design, which assigns participants in determined blocks to achieve proportions while at the same time randomly assigning.

The evaluation consortium ran training sessions for trial staff in winter 2017 and provided written guidance and videos. It was crucial that trial staff followed these to ensure ethical engagement of recruits, to protect integrity of the trials and the reliability of the results. Training and guidance covered: an overview of the evaluation and trial; accessing and operating the randomisation tool; assessing eligibility; giving information; gaining informed agreement to take part; baseline data collection; giving the randomisation result neutrally; and, providing information on next steps.

1.7. Referrals and the trial population

The initial intention was to generate referrals into the trials across 12 months, leading to around 14,100 people being recruited and randomised.

• In WMCA, this led to a target of c.6,600 with 50% (3,300) allocated to treatment (the IPS service) and 50% to the control group
• In SCR, the target was c.7,500 recruits, with 50% (3,750) allocated to the IPS service and 50% to the control group
  • Of these it was anticipated that 70% would be either unemployed and seeking work (SCR OOW, N=5,250) and 30% would be in employment but struggling/off-sick (SCR IW, N=2,250)

In each site, referral routes covered: primary care, community care (such as pain clinics and IAPT)^{[footnote 1]}: and self-referrals. Gaining referrals proved challenging (see sections 3.1 and 3.2) and so an extension of 6 months to the referral window was granted and the expectation for recruitment was reduced to c.4,500 overall in WMCA and c.6,600 across the SCR trial groups.

Figure 1.3 illustrates the inflow of recruits to the trial over its 18-month recruitment window. There was a notable increase in recruits in the final months of the recruitment window, with over 1,500 recruits randomised in October 2019. In total, 9,785 people were randomised, with recruits to each trial group shown in Table 1.1.

Table 1.1: Number of recruits randomised, by site and trial arm

	SCR	SCR	SCR	SCR	WMCA	WMCA
	IW	IW	OOW	OOW	OOW	OOW
	T	C	T	C	T	C	Total
Total recruits randomised	1,260	1,259	1,799	1,792	1,837	1,838	9,785

Source: Baseline data collection

1.3 Trial referral over time, with key dates

All recruits completed the baseline survey prior to randomisation. This showed a quarter (26%) were aged under 30, with 20% aged 30-39, 21% aged 40-49, and 33% aged over 50. This was broadly similar across the 3 trial groups (Table 1.2).

Table 1.2: Age of recruits, by trial group

Age	SCR IW %	SCR OOW %	WMCA OOW %	Total %
< 30	23	29	26	26
30 – 39	22	19	19	20
40 – 49	23	20	21	21
50+	32	32	34	33
Base	2,519	3,571	3,675	9,785

Source: Baseline data collection, all recruits answering this question

Over half (52%) of the recruits were male and 48% were female (Table 1.3). Recruits to the SCR IW group were more likely to be female (57%) than trial recruits overall.

Table 1.3: Gender of recruits, by trial group

Gender	SCR IW %	SCR OOW %	WMCA OOW %	Total %
Male	43	56	54	52
Female	57	44	46	48
Other	-	-	-	0
Base	2,517	3,585	3,671	9,773

Source: Baseline data collection, all recruits answering this question

Nearly 4 in 5 (79%) were from white ethnic backgrounds, with 10% from an Asian or Asian British background, 7% Black or Black British, 3% Mixed, and 2% from Other ethnic groups. Recruits in WMCA were more likely to be from minority ethnic groups (36%), than in SCR (10%), reflecting the ethnic profiles of the sites (Table 1.4).

Table 1.4: Ethnic group of recruits, by trial group

Ethnic group	SCR IW %	SCR OOW %	WMCA OOW %	Total %
White	90	86	64	79
Asian/Asian British	4	5	18	10
Black/African/Caribbean/Black British	3	4	12	7
Mixed/multiple ethnic groups	2	2	4	3
Other ethnic group	2	3	2	2
Base	2,510	3,578	3,652	9,740

Source: Baseline data collection, all recruits answering this question

All trial recruits were asked about their health conditions, illnesses, or impairments lasting 12 months or more and could list multiple conditions. Multi-morbidity (the simultaneous presence of 2 or more diseases or medical conditions) was prevalent. The most frequently cited health conditions were stress or anxiety (81%), depression (66%), fatigue or problems with concentration or memory (63%), and pain or discomfort (57%) (Table 1.5).

Table 1.5: Health conditions, illnesses or impairments lasting 12 months or more, by trial group

Health condition	SCR IW %	SCR OOW %	WMCA OOW %	Total %
Stress or anxiety	86	81	76	81
Depression	68	68	64	66
Fatigue or problems with concentration or memory	72	61	60	63
Pain or discomfort	57	55	59	57
Problems with neck or back	40	37	41	39
Problems with legs or feet	36	37	40	38
Dizziness or balance problems	33	32	34	33
Problems with arms or hands	26	25	28	26
Problems with bowels, stomach, liver, kidneys or digestion	27	24	24	25
Chest or breathing problems	24	24	25	24
Arthritis	20	20	23	22
Skin conditions or allergies	23	21	20	21
Heart or blood pressure problems	19	20	24	21
Learning difficulties	11	20	21	18
Mental health condition (other than depression/stress)	12	15	15	14
Other health or disability issue	12	13	15	14
Difficulty with seeing	11	12	17	13
Difficulty with hearing	11	10	11	11
Problems due to alcohol or drug addiction	4	9	6	7
Speech problems	6	7	8	7
Progressive illness not covered above	7	7	7	7
Don’t know/Prefer not to say	1	1	1	1
Base	2,519	3,591	3,674	9,784

Source: Baseline data collection, all recruits answering this question

Recruits were also asked the extent to which their health condition(s) or disability(ies) limited their ability to carry out everyday activities. A third (33%) said a great deal, 44% said to some extent, 18% a little and 4% not at all. There were no marked differences between trial groups (Table 1.6).

Table 1.6: Extent that health condition or disability limits ability to carry out everyday activities, by trial group

Extent	SCR IW %	SCR OOW %	WMCA OOW %	Total %
A great deal	39	33	29	33
To some extent	42	45	44	44
A little	16	18	21	18
Not at all	3	4	6	4
I do not have a health condition or disability	0	0	0	0
Base	2,504	3,547	3,653	9,704

Source: Baseline data collection, all recruits answering this question

On recent employment history, a majority of the SCR IW group (76%) had been in employment throughout the 2 years pre-randomisation. In the OOW groups, 50% in SCR and 59% in WMCA had not worked throughout the 2 years prior to randomisation (Table 1.7).

Table 1.7: Employment history in the last 2 years, by trial group

Employment history	SCR IW %	SCR OOW %	WMCA OOW %	Total %
Always been in paid employment	76	5	3	22
Been in paid employment for more than half the time	14	20	16	17
Been in paid employment for less than half the time	10	25	22	20
Not been in paid employment in last 2 years	0	50	59	40
Base	2,500	3,549	3,655	9,704

Source: Baseline data collection, all recruits answering this question

Where recruits had not been in employment continuously over the last 2 years, the most frequently cited reasons were mental health (54%), being unable to find a suitable job (which could reflect issues such as lack of local opportunities, lack of skills and experience, lack of flexible working or accommodation of health conditions) (52%), and physical health issues (43%) (Table 1.8).

Table 1.8: Reasons not in paid employment in the last 2 years, by trial group

Reason	SCR IW %	SCR OOW %	WMCA OOW %	Total %
Mental health issues	56	57	52	54
Unable to find a suitable job	43	51	55	52
Physical health issues	31	42	45	43
Caring responsibilities	15	20	21	20
Education/training	17	15	15	15
Other	13	11	12	12
Don’t know/Prefer not to say	1	1	1	1
Base	614	3,378	3,527	7,519

Source: Baseline data collection, all recruits answering this question

All recruits were asked what had made it difficult for them to find work. Difficulty finding a suitable job was most commonly cited (77%); followed by mental health conditions (62%); lack of confidence in skills or abilities (57%); lack of qualifications or experience (52%); and a physical health condition (49%) (Table 1.9).

Table 1.9: Barriers to employment at initial appointment, by trial group

Barrier	SCR IW %	SCR OOW %	WMCA OOW %	Total %
Difficulty finding a suitable job	59	82	83	77
Mental health condition	64	65	58	62
Lack of confidence in abilities or skills	53	58	58	57
Lack of qualifications or experience	41	55	57	52
Physical health condition	44	49	53	49
Availability or cost of transport to work	26	38	38	35
Being financially being worse off	34	27	27	29
Caring for a child, or an elderly or disabled family member	16	18	17	17
Another reason	12	11	12	11
Availability or cost of childcare	9	9	10	9
Nothing	2	1	0	1
Not applicable	2	0	0	1
Don’t know/Prefer not to say	1	0	0	0
Base	2,519	3,591	3,675	9,785

Source: Baseline data collection, all recruits answering this question

Nearly 3 in 10 recruits (29%) had no or low-level formal qualifications, around half were qualified at Level 2 or 3 (48%), and 23% had higher level qualifications (Table 1.10). 76% of recruits had dependent children under the age of 16, while 24% did not (Table 1.11).

Table 1.10: Highest level of qualification, by trial group

Qualification	SCR IW %	SCR OOW %	WMCA OOW %	Total %
Degree/higher degree/equivalent; NVQ or SVQ levels 4 or 5	26	15	13	17
Higher educational qualification below degree level	7	7	6	6
A levels or Highers; NVQ or SVQ level 3	25	17	17	19
GCSE grades A-C or equivalent	24	29	32	29
GCSE grades D-G or equivalent	9	14	13	12
Other quals inc. vocational and foreign quals below degree level	3	6	5	5
No formal qualifications	6	14	15	12
Base	2,481	3,505	3,609	9,595

Source: Baseline data collection, all recruits answering this question

Table 1.11: Whether recruits had dependent children, by trial group

	SCR IW %	SCR OOW %	WMCA OOW %	Total %
Dependent children	73	77	76	76
No dependent children	27	23	24	24
Base	2,511	3,579	3,666	9,756

Source: Baseline data collection, all recruits answering this question

1.8. IPS fidelity in the trials

The IPS-25 scale (Becker et al, 2019) measures fidelity and has been shown to have good psychometric properties, including predictive validity – which means the extent to which a score on a scale or test predicts scores on agreed measure(s) (see Bond et al., 2012). The rationale for using fidelity scales to guide implementation is that interventions successfully replicating core principles of IPS will achieve similar outcomes to those found in the evidence base that establish IPS effectiveness (that is, interventions with higher IPS fidelity have better outcomes) (see Becker et al, 2019, Kim et al, 2015). The HLT design created 2 exceptions to these principles:

• trial inclusion/exclusion criteria and randomisation meant that not everyone who wanted to work was eligible for IPS support
• the IPS-LITE service model meant support was time-limited; recruits could access 9 months of support before starting/recommencing a job, and 4 months of in-work support after starting a job

The sites were responsible for assessing the fidelity of their IPS services. Fidelity reviews were undertaken by Social Finance in WMCA towards the start of delivery in August 2018 and a year later in July/August 2019. Social Finance then undertook fidelity reviews of all trial providers across WMCA and SCR in autumn 2019. The scores ranged from a ‘fair’ degree of IPS fidelity (in the range of 74-99 points scored) to ‘good’ IPS fidelity (in the range of 100-114 points scored).^{[footnote 2]}

The 2019 reviews noted common items amongst all trial providers where IPS fidelity was scored lower:

• employer engagement and job development, where there was a reliance on applying to online opportunities rather than relationship development and accessing the hidden jobs market
• integration and links with health professionals
• lack of regular discussion with the treatment group about the impact of disclosure to employers about health conditions

1.9. Headline findings

1.9.1. Impact and economic evaluation findings

The 12-month impact analysis showed that receipt of the IPS services made a significant difference to the experience of the treatment group in both trial sites. However, the nature of impact differed by site and by trial group.

• In SCR, the results differed by trial group:
  • For the SCR IW trial group, being assigned to the treatment group increased the probability of having been in work for 13 or more weeks in the year following randomisation by 3 percentage points (ppts) at the 90% significance level, that is, below the conventional 95% level. There was a small positive impact on health (0.10 standard deviations (sd)^{[footnote 3]} significant at the 90% confidence level. Impact on wellbeing was substantial (0.18 sd) and strongly significant (99% confidence level).
  • Assignment to the SCR OOW treatment group had no statistically significant effect on employment but positive small impacts on health and wellbeing (0.10 and 0.12 sd respectively) significant at 90% level.
  • When considered as a whole (All SCR) no employment impact was detected but impacts on health and wellbeing were significant at the 99% level, despite being small (0.10 and 0.14 sd, respectively).
• In WMCA, where all recruits were OOW, 18% of the control group and 22% of the treatment group saw employment outcomes. This difference at 4 ppt was significant at the 99% statistical confidence level. However, in WMCA there was no discernible impact on either health or wellbeing.
  • The employment impact in WMCA was substantial – meaning that recruits in the treatment group were 20% more likely to find work than those receiving business-as-usual (BAU) support. Additionally, the economic evaluation showed this employment impact was sustained at the point 21 months following randomisation and statistically significant at the 95% confidence level.
• When combined, no impact was detected on employment for the 2 OOW trial groups (All OOW) but small impacts on health and wellbeing were statistically significant (0.08 and 0.10 sd, respectively) at the 95% significance level.
• No impact was found on earnings for any group or in either site.
• The economic evaluation found that the costs of delivering the IPS services varied considerably between trial groups and the 2 sites.
  • The spend per member of the SCR IW treatment group was £2,116.00.
  • For the SCR OOW treatment group, it was £2,416.00 per person.
  • The spend per member of the All SCR treatment group was £2,292.00.
  • The spend per recruit to the treatment group in WMCA was £3,893.00. This resulted from the lower number of recruits in WMCA (1,837) compared to SCR (3,059 All SCR).
  • This meant that total costs for those in the All OOW treatment group were also higher at £3,162.00 per person.
• The economic evaluation also estimated the economic benefits derived from improvements in health and the financial benefits deriving from improvements in employment. This showed that:
  • The financial benefits from employment impacts (given the lack of effect on earnings), were not much greater than the costs of delivering the IPS services. For every £1 spent on the IPS services, a financial return of 2p resulted for the SCR IW trial group, and of 1p for WMCA.
  • Due to the health impacts, economic returns were greater. Every £1 spent on the IPS services delivered returns of £2.32 for the SCR IW trial group, £2.02 for SCR OOW, and £1.22 for All OOW.
  • Sensitivity analysis was used to quantify the statistical uncertainty around the impact estimates and therefore the benefit-to-cost ratios. This showed these were subject to significant variation and meant it was not possible to state definitively that the same results would emerge if the trial were to be re-run.
• Overall, the economic findings indicated that to generate better returns on investment, achieving health-related outcomes alongside employment outcomes from the IPS services was necessary.
• When longer-term employment outcomes (21 months post-randomisation) in the linked evaluation data set were examined, an impact on earnings was seen in WMCA shortly following the 12-month outcome measure documented by the statistical analysis plan. This may suggest financial returns are underestimated.

1.10. Accounting for differences in outcomes

The evaluation evidence was examined for factors that might be driving the differences in impact. This included searching for any effect from the COVID-19 pandemic. Some additional analyses using labour market information (LMI) showed the labour markets in WMCA and SCR were very different and recovered in different ways from the pandemic. There was a possibility that the IPS service made a greater difference on employment outcomes in a weaker labour market with low rates of employment (WMCA) than it did in a more buoyant labour market (SCR). However, this could not be stated definitively as the LMI could not be linked to the evaluation data set in any meaningful way.

Potentially supporting the emergence of health outcomes, qualitative evaluation evidence showed a greater connection between the health system and trial in SCR than in WMCA. This stemmed from local Clinical Commissioning Groups and Local Authorities collaborating on design, whereas in WMCA this was contracted out. The survey showed that recruits generally (that is, treatment and control) in SCR made greater use of health support organisations than those in WMCA. MI showed that the SCR IPS service offered longer duration but less frequent meetings than in WMCA which may have been conducive to discussing health and wellbeing and agreeing follow-up actions and referrals. In contrast, the MI showed that the IPS service in WMCA offered more frequent meetings and enrolled the treatment group into the IPS service more rapidly which meant jobsearch could also commence more quickly. The MI also showed a mix of face-to-face and telephone check-in interactions in WMCA which may have increased momentum on jobsearch.

Economic data suggested that caseloads in WMCA were smaller than in SCR. The qualitative data showed that the mixed IW and OOW caseload in SCR brought greater complexity and different practical needs that were challenging for employment specialists to manage. The qualitative and economic evidence suggested more focus on employer engagement in WMCA than in SCR, potentially due to a smaller and less complex caseload.

1.11. Lessons for the disability employment gap

The gap in employment rates between disabled and non-disabled people has been a longstanding policy challenge and the reason the trials were introduced. The tight labour market that has emerged since the pandemic suggests employers need to access a wider talent pool to fill vacancies, which may improve conditions for the employment of people with LTHC. A number of lessons can be drawn from the trials.

The evidence shows that improving health condition management and achieving health referrals alongside employment support built people’s capability and self-belief. The strengths-based approach brought by the IPS services was an important facilitator of confidence in job search capability and in feeling that work is possible. The treatment group appreciated being able to focus on job roles that were matched to their goals and capabilities.

Employment specialists’ engagement with employers, and taking a holistic view to obstacles to the labour market, helped to increase employment outcomes. Overall, the evidence suggested a need for greater differentiation, and exploration of intersectionality in the design of employment support programmes, to ensure that people from all backgrounds find the support suitable to their needs.

Identifying hidden vacancies and supporting job development was productive. There was evidence of employer and organisational confidence in employing people with LTHC increasing as a result of the IPS support received.

1.12. Lessons learned for delivery

A number of the factors that appeared explanatory for the differences in outcomes also appear in the IPS Fidelity Scale, where a higher overall score typically correlates strongly with impact on employment. Caseload is an important, determining factor in respect of time that can be spent on other activities. Higher caseloads in SCR may have meant employment specialists had less time to spend on members of the treatment group, compared with WMCA, which may have impacted on the nature and extent of employment support. Higher caseloads may have constrained the amount of employer engagement, which was less in SCR. Mixed OOW and IW caseloads in SCR added complexity as the 2 groups had different preferences and differing needs. Future implementation might benefit from smaller, specialised caseloads.

Ensuring time for employer engagement is crucial in future delivery. In both sites, employment specialists said this task was difficult, and those with less prior experience wanted more training and support on this. It was also noted that a key barrier for the recruitment of people with health conditions was employers’ limited understanding of the cost of reasonable adjustments and how the government could support these. Information sharing and time spent in discussion with employers could overcome these concerns. In SCR, employer workshops on supporting staff with mental health conditions were helpful in creating impetus on this agenda.

More generally, as it is an employment service, rather than an intervention focused on health, it is possible that a focus on health outcomes in the SCR IPS service took priority over employment outcomes, shifting the service from a ‘place then train’ to a ‘train then place’ approach for some in the treatment group. However, there was also evidence suggesting the SCR IPS service aimed for better quality work than business as usual, which if achieved should lead to better health outcomes given the social gradient within employment identified by Marmot (2020). The 12-month survey showed that the SCR OOW control group felt their employment was more precarious than the SCR OOW treatment group, feeling, for example, that work makes it harder to manage their health (44% compared with 27% in the treatment group). This focus should not be lost for future delivery if the therapeutic value of work is to be tested.

2. Main findings

2.1. The impact of the HLTs on recruits

The primary outcomes selected for the HLTs to measure effects in the 12 months following randomisation covered:

• employment – whether employed for 13 or more weeks in the 12 months following randomisation (based on HMRC PAYE RTI data) – a measure aligned with public policy measures to help individuals enter the labour market
• earnings – total earnings in the 12 months following randomisation (based on HMRC PAYE RTI data)
• health – as measured by the EQ5D5L instrument administered as part of the 12-month survey
• wellbeing – as measured by the SWEMWBS instrument as part of the 12-month survey

The results of the impact analysis are summarised in Table 2.1 below.

Table 2.1: Summary of the impacts obtained for the trials

	Employment	Earnings	Health	Wellbeing
SCR IW	3ppt *	£442	0.10 sd *	0.18 sd ***
SCR OOW	-2ppt	-£233	0.10 sd *	0.12 sd *
All SCR	1ppt	£102	0.10 sd ***	0.14 sd ***
WMCA	4ppt ***	£150	0.05 sd	0.9sd
All OOW	1ppt	-£51	0.08 sd **	0.10 sd **

Bold indicates impact observed; asterisks indicate level of confidence/significance associated with observed impacts as follows: * 90%; ** 95%; *** 99%. n/c – not calculated

Source: Final evaluation data set

The 12-month impact analysis showed that receipt of the IPS services made a significant difference to the experience of the treatment group in both trial sites. However, the nature of this impact differed by site, with WMCA showing strongly significant employment impact but no effect on health and wellbeing. In contrast, in SCR the results were more weighted, overall, towards health and wellbeing outcomes. A small employment impact was seen for the SCR IW group but this was below the conventional level of confidence. No significant effect was found in either site on earnings at the 12-month outcome point although the trend was positive for the SCR IW and WMCA trial groups.

• In SCR, the results differed by trial group:
  • For the SCR IW trial group, being assigned to the treatment group increased the probability of having been in work for 13 or more weeks in the year following randomisation by 3 ppts at the 90% significance level, that is, below the conventional 95% level. There was a small positive impact on health (0.10 sd)^{[footnote 3]} significant at the 90% confidence level. Impact on wellbeing was substantial (0.18 sd) and strongly significant (99% confidence level).
  • In the SCR OOW group, 27% of the control group and 25% of the treatment group were in work at the 12-month point; a 2 ppt difference with business-as-usual achieving more of these outcomes – that is, being assigned to the treatment group had no statistically significant effect on employment. However, positive but small impacts were seen for health and wellbeing (0.10 and 0.12 sd respectively), which were significant at the 90% level.
  • When considered as a whole (All SCR) no employment impact was detected but impacts on health and wellbeing were significant at the 99% level, despite being small (0.10 and 0.14 sd, respectively).
• In WMCA, where all recruits were OOW on joining the trial, 18% of the control group and 22% of the treatment group saw employment outcomes. The difference in these outcomes of 4 ppt was significant at the 99% statistical confidence level. However, in WMCA, while a positive trend was seen, there was no discernible impact on either health or wellbeing.
  • The employment impact in WMCA was substantial at 4 ppt – meaning that recruits in the treatment group were 20% more likely to find work than those receiving business-as-usual (BAU) support.
  • Additionally, analysis for the economic evaluation showed this impact to be sustained at a point 21 months following randomisation. This was statistically significant at the 95% confidence level.
• When the 2 OOW trial groups were combined (that is, All OOW which combines all WMCA recruits, and the SCR OOW recruits) the analysis showed no impact on employment. However, impacts on health and wellbeing were statistically significant (0.08 and 0.10 sd, respectively) at the 95% significance level.

The results of the economic analysis are summarised in Table 2.2 below.

Table 2.2: Summary of the costs and returns obtained for the trials

	Costs per recruit to treatment group	Financial return for every £1 spent	Economic return for every £1 spent
SCR IW	£2,116	£0.02	£2.32
SCR OOW	£2,416	£0	£2.02
All SCR	£2,292	n/c	n/c
WMCA	£3,893	£0.01	£0
All OOW	£3,162	£0	£1.22

Bold indicates impact observed; asterisks indicate level of confidence/significance associated with observed impacts as follows: * 90%; ** 95%; *** 99%. n/c – not calculated

Source: Final evaluation data set

• The economic evaluation found that the costs of delivering the IPS services varied considerably between trial groups and the 2 sites.
  • the spend per recruit to the SCR IW treatment group was £2,116.00
  • per recruit to the SCR OOW treatment group the spend was £2,416.00
  • this led to a per person spend of £2,292.00 for the treatment group across SCR; that is, All SCR
  • the spend per recruit to the treatment group was substantially higher in WMCA at £3,893.00 per person
  • the reason for these higher costs was the lower number of recruits in WMCA (1,837) compared to SCR (3,059; covering All SCR)
  • this meant that total costs for those in the All OOW treatment group were also higher at £3,162.00 per person

The economic evaluation estimated that the economic benefits derived from improvements in health were worth more to society and the exchequer than the financial benefits deriving from the improvements in employment, which additionally were constrained by the limited earnings effect in either trial.^{[footnote 4]}

The estimates showed that the financial benefits from the employment impacts were not much greater than the financial costs of delivering the IPS services. The IPS service was also more expensive to deliver in WMCA than it was in SCR, which is material as in WMCA a strong and substantial impact on employment was observed. For these reasons, the economic analysis found a very small net financial return to the IPS service, as follows: for every £1 spent on the IPS services, a financial return of 2p resulted for the SCR IW trial group, and a return of 1p resulted for WMCA. No financial return was observed for either the SCR OOW, or All OOW trial groups because no employment impact was observed at the time of measurement which was 12 months following randomisation or found in the administrative data where outcomes could be tracked up to 22 months after randomisation.

In contrast, due to health outcomes being achieved, initial analysis of the economic benefits from the IPS services suggested that every £1 spent on the IPS service delivered £2.32 of benefits for the SCR IW trial group, £2.02 for the SCR OOW trial group, and £1.22 of benefits for the All OOW trial group. The latter return was observed because when the OOW groups were pooled, a significant effect on health outcomes was detected. This was due to the combination of the impact seen in the SCR OOW trial group with the positive trend in health in WMCA.

However, while initially the economic analysis appeared to make the case that a positive net benefit resulted from the investment in the IPS service in SCR, due to the health impacts, a sensitivity analysis (in line with other analysis using Monte Carlo simulation) found statistical uncertainty around the benefit-to-cost ratios which meant it was not possible to state definitively that the same results would emerge if the trial were to be re-run. The Monte Carlo simulation randomly selects values from the probability distributions of each impact estimate, based on their standard errors 10,000 times. These distributions then simulate the distribution of benefit-cost ratios. Overall, the economic findings suggested that to generate economic benefits to society, achieving health-related outcomes alongside employment outcomes from the IPS services was necessary.

Finally, when the longer-term outcomes recorded by administrative data (at 22 months following randomisation) in the linked evaluation data set were reviewed for evidence of sustained employment, an impact on earnings was observed for WMCA shortly following the 12-month measure identified within the statistical analysis plan (SAP). This may suggest that financial returns are underestimated for WMCA in the economic evaluation.

2.1.1. Putting the impacts into context

The existing evidence base demonstrates the employment outcome used within IPS studies varies. The most common measure internationally is of competitive employment in the open jobs market of at least 1 day in the follow-up (post-placement) period (Burns et al., 2015). Given the more ambitious definition of a sustained employment outcome (13 weeks in employment), the results from HLTs are lower than the mean competitive employment rate of 55% for IPS and 25% for controls across 28 trials for people with severe mental illness (IPS Employment Centre, 2021).

It is also worth noting that as IPS is primarily an employment intervention integrated with health services, most other sources focus on employment as the primary outcome and may only explore mental health, including mental health and wellbeing, as secondary outcomes. Again, this differentiates the HLTs as employment, health and wellbeing were equal as primary outcome measures.

3. Explaining the impacts

A key question is why these differences in impact across outcome measures between trial sites emerged. The impact evaluation explored a number of theories, but not all could be investigated empirically. This analysis showed:

• The difference in health and wellbeing outcomes for the OOW groups might be explained by differences in the content of support. Service provider MI included information on the configuration of support (frequency and duration of IPS meetings) with indications of some differences between the sites. However, it was not possible to correlate these to the differential health and wellbeing impacts. Differences in the observable characteristics of the trial populations in each site were explored but did not account for the different health impacts.
• A couple of theories were likely to explain the difference in employment impacts, with compositional differences again discounted. First, the employment rate was generally higher in SCR than in WMCA across the trial period, suggesting labour market differences were important to IPS outcomes. The second centred on whether support received by the SCR OOW control group was instrumental.
• When the evaluation evidence was considered in the round, a third theory emerged that differences related to elements of IPS Fidelity may have been important to the employment impacts.

As such, a few lines of enquiry were developed for the synthesis analysis. Much of this focuses on the 2 OOW trial groups since it is the difference in their outcomes that is most puzzling. However, this does not mean the impacts and returns seen for the SCR IW group are discounted; the health and employment outcomes seen for this group alongside the tentative, net economic returns suggests that taking forward a further IPS service targeted this group could be valuable.

In the following sections, a boxed summary of key information precedes more detailed analysis.

3.1. Site level factors that might account for differences in observed health impacts

Throughout the trial surveys, health was the most commonly cited barrier to work and the descriptive analysis of the survey suggested that reducing recruits’ health barriers led to an increased capability and capacity to work. Despite this, health impacts were observed in SCR but not in WMCA.

• At baseline and at 12 months following randomisation, the SCR trial groups indicated that their mental health formed the greatest barrier to working, whereas the WMCA trial group more commonly cited physical health.
• The process evaluation and Context-Mechanism-Outcome (CMO) analysis found some evidence that the health system and IPS service were better linked in SCR than in WMCA stemming from strategic-level collaboration to design the trial.
• As IPS is a voluntary employment support provision, improvements in health would result from engagement with health services. The closer linkage to the health system in SCR – demonstrated by referral routes and greater likelihood of recruits to be engaging with health support – might lead to this.
• This was supported by 12-month survey evidence showing the SCR treatment groups made better use of health services than SCR control groups. This was not seen in WMCA. The IPS service may have enabled this capability in SCR.

For all recruits (treatment and control groups), the surveys found that health was commonly cited as the most important barrier to work. This suggests the importance of addressing health problems and improving condition management in parallel to providing employment support to secure work.

The nature of the health condition that posed the greatest barrier to work in the 12-month survey differed between sites with mental health predominating in SCR IW (23%) and SCR OOW (24%) compared with WMCA at 19% (see Table 6.8) 12-month survey report, Appendix A: Table 56). This reflected differences when recruits joined the trials: in SCR, recruits were 6-8 ppt more likely than those in WMCA to report mental health conditions.

The 12-month survey found that reducing recruits’ health barriers led to an increased capability and capacity to work. Alongside this, there was evidence that recruits (treatment and control) in SCR were better connected with the health system and made more use of health support than recruits in WMCA. This may have stemmed from the referral routes in operation in each site:

• SCR recorded more direct referrals from health settings: 42% in total (with 18% from a GP, and 24% from specialist care service)
• in WMCA, 20% of referrals came directly from health service providers (16% from a GP, and 4% from a specialist care service respectively)

These may have resulted from the early strategic collaboration between health partners in the design of the SCR trial, whereas this process was outsourced in WMCA. On this basis, it could be said that recruits in SCR were more likely than those in WMCA to be actively engaging with health services at the point of referral.

The 12-month survey also found variation in recruits’ usage of wider support between the 2 sites 12 months after randomisation. It captured use of employment and health-related support services (see Table 6.4 in the Appendix to this report).

• SCR OOW recruits were more likely to have accessed support from a GP or other primary care service than those in WMCA (47% vs 40%, respectively)
• Recruits in WMCA were more likely to have accessed support from Jobcentre Plus than the SCR OOW recruits (49% compared to 39%)

These links between the health system and IPS service may have been an important factor in health impacts being achieved in SCR, not least as IPS is an employment service. It would be expected that health improvements would stem from engaging with health sources.

Exploring this further, in SCR both IW and OOW treatment groups were more likely than the control groups to report that accessing wider health support helped them to manage their health condition or disability.

• The SCR IW treatment group was more likely than the control group to say that the wider support helped them to manage their health condition (76% vs 63%)
• The SCR OOW treatment group accessing wider support services was more likely than those accessing wider support in the control group to say that this had positively affected health condition management (71% vs 60%)
• There was no significant difference between the treatment and control group respondents in WMCA who accessed health support saying that it positively helped condition management (64% vs 63%)

This suggests that the IPS service in SCR enabled recruits to make best use of wider services to manage their health conditions. This was supported to a degree in the survey analysis. The proportion of respondents (treatment and control) saying that wider support had positively affected health condition management was greatest for the SCR IW group (70%) with a more marginal difference between SCR OOW (65%) and WMCA (63%). It could be argued that greater connectivity between the IPS service and health support in SCR may have led to better health condition management and the health impacts that were observed.

3.2. Site level factors that might account for employment impact differences

Employment impact was observed in WMCA but not for the SCR OOW group, where the best contrast exists between the populations of the 2 trials.

• As the trials’ designs did not intend referrals to be made from Jobcentres and employment support organisations, these categories were not included specifically in the MI but in an ‘other’ category. The rate of referral from other sources was substantially higher in WMCA compared to SCR.
• However, recruits’ receipt of support from Jobcentre and other employment services was captured in the surveys, with the 12-month survey showing a substantially higher proportion of recruits in receipt of this in WMCA compared to SCR.
• However, as motivation levels to find work did not vary by the type of wider support received, the differences in employment impact could not be linked to this. In sum, there was limited evidence of site level factors affecting employment impacts.

While there were differences between the sites in referral routes for the OOW trial groups, these do not make a strong case for the difference in employment impacts. In parallel to comparatively high levels of referral from healthcare services in SCR, there were lower levels from ‘other’ sources including employment services (56%). In contrast, in WMCA, 80% of referrals were made by these other sources.

However, differences in accessing ongoing support from Jobcentre Plus were quite substantial between the sites (Table 6.4 in the Appendix). The final survey showed that 39% of the SCR trial group (treatment and control) were accessing support from Jobcentre Plus, compared to 49% in WMCA. When other employment services were added in, this gap increased substantially. This was due to there being more interacting employment programmes in WMCA than in SCR. On top of national programmes such as the Work and Health Programme there was also access to employment support funded by Big Lottery, an employment programme offered by a health trust, and employment programmes offered by the combined authority.

It may be that the differing combination of IPS with health in SCR or employment support in WMCA led to the differences in health and employment outcomes.

However, the proportion of 12-month survey respondents (treatment and control recruits) stating that the wider support they accessed positively affected their motivation to work did not significantly differ between the sites or trial groups. Hence, whether wider support was accessed through Jobcentre Plus or healthcare did not appear material to motivation to work. This is worth noting in light of information from the 2015 ESA Reform Trials, which found that motivation was a key determinant of employment outcomes for claimants with experience of long-term inactivity returned to Jobcentre from the Work Programme (Moran, 2017; Newton and Sainsbury, 2017).

Using the 12-month survey , controlling for the influence of recruits’ observed characteristics, the difference in outcomes between SCR OOW and WMCA remained substantial. Overall, the evidence emphasised a need to explore whether differences in the employment impact were driven by differences in IPS service delivery (section 2.2.2) or the ‘business as usual’ support available to the control group in SCR (section 2.2.3).

3.3. How differences in the IPS services may have affected the outcomes

Differences between the sites in the frequency and duration of IPS support to the treatment group may have influenced both health and employment outcomes.

• Caseload size is a key feature of IPS that is measured in IPS Fidelity Reviews. A caseload between 20 and 25 clients per full-time IPS employment specialist is common within IPS trials. While the available evidence suggested that in both sites, caseloads were in this range, those in SCR tended to be at the higher end, whereas those in WMCA sat at the lower end. This was likely to have implications for time spent on other IPS activities beyond meeting clients.
• The mixed caseload in SCR may have had an effect. In the process evaluation, employment specialists noted that the SCR IW and OOW trial groups had different preferences for meeting times and schedules, and differing needs. This may have constrained resources for other activities and affected support quality.
• Employer engagement is a core aspect of IPS support, opening the potential to ‘tap in’ to job opportunities within ‘the hidden job market’. Employment specialists in the trial with a background in health could find this challenging.
• There was some evidence that employment specialists in WMCA dedicated more time to employer engagement than those in SCR. This may reflect an opportunity crated by smaller, single-focused caseloads.
• Service provider MI showed a different pattern of meetings for the OOW treatment groups. In WMCA, there were more meetings, and a mix of short and longer duration meetings. Meetings in SCR were longer and less frequent.
• These different patterns may have had different benefits. Longer duration, less frequent meetings may have allowed for more discussion of progress on health and wellbeing; short, more frequent meetings may have better supported ongoing job-search.
• The WMCA service more rapidly enrolled the treatment group into the IPS service, which enabled job search to commence more quickly. This is a principle of IPS and therefore important to employment outcomes.

Using IPS fidelity principles, the differences in service delivery are potentially explanatory. Within the existing literature, causality exists between IPS services achieving high fidelity and impact on employment.

• The higher and mixed caseloads in SCR suggest an increased level of pressure on employment specialists and less time to deliver individualised support compared to a lower, more homogenous caseload seen in WMCA.
• Lower caseloads may also have allowed for the increased frequency of contact seen in WMCA which may have kept the momentum of the job search going. In turn, this may have led to fewer in WMCA disengaging from support. It may also have created more time for liaising with employers to source vacancies. Alongside this, the quicker start in WMCA may have capitalised on the impetus of signing up to the service on the basis of wanting to find work.

3.3.1. Differences in caseload size and make-up

Within the IPS principles, the maximum caseload for any full-time employment specialist is recommended to be 20 or fewer clients (Becker et al., 2019). Caseload size is a key feature of IPS support, measured as part of Fidelity Reviews. However, in practice, caseloads in other IPS trials have varied between 20 and 25 clients per full-time IPS employment specialist (see, for example, Burns et al., 2015; Swanson et al., 2008 cited in: Fergusson et al., 2012; Burns & Catty, 2008). An average caseload of 30 clients per IPS employment specialist was reported by Reme et al. (2015) where the trial included people in work.

At the point of commissioning the HLTs, it was anticipated that the maximum open caseload for any full-time HLT employment specialist would be 25-30 in the first year, but could rise to 30-35 as the service matured.

Finding precise measures of caseload size in the evaluation data was challenging due to the varying levels of referrals and take-up over the course of the 30-month delivery period, and fluctuating levels of staffing. However, the fidelity reviews indicated that both sites operated within the intended caseload size for IPS ranging between 10 and 26, although employment specialists in SCR worked towards the upper range of this on average, while those in WMCA they worked towards the lower end. Intuitively, a lower caseload would allow more time to be spent on each individual supporting their employment goals.

Confirming this, the process evaluation found that employment specialists with higher caseloads reported finding their workload difficult to manage, making it challenging to fully implement IPS. The higher caseloads in SCR may have meant employment specialists had less time to spend on members of the treatment group, compared to WMCA. The finding from the economic evaluation that the service was more expensive to deliver in WMCA may be pertinent on this point.

Moreover, in SCR employment specialists found it challenging to manage mixed IW and OOW client lists, as many of the IW group required evening and weekend meetings while the OOW treatment group preferred daytime meetings. Furthermore, the process evaluation findings document a concern from an employment specialist in SCR that their higher caseload might be leading to a lower quality of support (see section 1.3 of the Implementation and 4-month outcomes report). It is impossible to say how common this feeling was based on the data available.

Nonetheless, satisfaction with the IPS service was typically higher in SCR than in WMCA (see 12-month survey descriptives report) which implies that those receiving the service felt it met their needs.

3.3.2. Differences in the effectiveness of employer engagement and job brokerage

Employer engagement is a core aspect of delivering IPS support, opening the potential to ‘tap in’ to job opportunities within ‘the hidden job market’. Each employment specialist should make at least 6 face-to-face employer contacts per week on behalf of clients looking for work (Becker et al., 2019).

The implementation research indicated that employment specialists, typically those from a health background, found employer engagement challenging. Those who had previously worked in employment support were more confident and described a wider range of networking strategies. In WMCA, the IPS service was delivered by employment support organisations in 2 geographic areas implying greater confidence on this agenda.

However, the implementation research indicated the most significant perceived barrier to employer engagement was a lack of time; when caseloads were high, employment specialists were not able to prioritise this. This is pertinent as SCR caseloads tended to be higher on average than in WMCA. Trial staff reported that employer engagement became more of a focus once the recruitment and referral for the trial was completed which allowed more time for this activity.

Information from the economic evaluation data suggested that SCR employment specialists spent less on employer liaison than those in WMCA. These data were gathered in a staff survey towards the end of delivery so are indicative only, and suggested c30% more time was spent on average liaising with employers in WMCA compared to SCR.

Overall, implementation evidence suggests that WMCA had more effective employer engagement possibly because they had more time to dedicate to this activity.

3.3.3. Difference in the intensity of treatment and proportion of recruits leaving treatment early

The impact evaluation found that across the trials, recruits to the treatment group had an average of about 12 face-to-face sessions or telephone contacts. This was higher in WMCA (mean = 14) than in SCR (c10 for IW and OOW groups). However, restricting the analysis to only IPS sessions lasting more than 15 minutes – to enable a focus on meaningful, real-time personal support in line with the IPS model – revealed that the mean number of longer sessions in WMCA (about 8) was less than that in SCR (about 10 for both IW and OOW groups). This suggested a relatively greater use of short catch-up and check-in sessions in WMCA than in SCR, where longer, more in-depth sessions were more common – this latter format, may have worked well for health and wellbeing support. In contrast, short check-ins and catch-ups may have kept up the momentum on job search activity.

The process evaluation found that in SCR, some interviewees had difficulties completing job forms and applications, and may have benefited from more frequent support. The model of IPS support with more frequent touch points in WMCA might have proved more effective at working towards employment outcomes.

A further point is that the treatment group began their IPS support more quickly in WMCA than in SCR with a third (31%) of the treatment group experiencing their first session within a week of randomisation. This may have led to quicker activation in job search; a key principle of IPS that might help explain employment outcomes (Becker et al, 2019). A slower schedule of meetings, given the time needed to lead vocational profiling, may have delayed this contact, and meant that employment outcomes fell below those of more rapid, responsive, and proactive IPS services.

It is also worth considering that IPS is an employment support service rather than an intervention focused on health support. In explaining the difference in outcomes between the two sites, it is possible that the focus on health support including health referrals and condition management in SCR took priority over employment outcomes, shifting the service from a ‘place then train’ model to a ‘train then place’ approach at least for some in the treatment group. However, some evidence suggested the IPS service in SCR aimied for better quality work than “business as usual” support. For example, the 12-month survey showed that the SCR OOW control group felt their employment was more precarious than the SCR OOW treatment group, feeling for example, that work makes it harder to manage their health (44% compared with 27% in the treatment group). Based on Marmot’s assessment in 2020, better quality work would deliver better social and economic benefits to the individual and society.

3.4. Treatment and control differences affecting employment outcomes

The employment rates between the two control OOW groups were markedly different with SCR control group being 9ppt more likely to be in employment than those in WMCA. This gap was greater than seen in the rates of employment for the OOW treatment groups with SCR showing 3ppt greater likelihood of employment than WMCA treatment group.

• The OOW control groups in both sites showed greater likelihood than the treatment group to use employment support, for instance from Jobcentre Plus, and take-up of health support was similar between sites for the control groups.
• However, the OOW treatment group in SCR used health support to a far greater degree (11ppt) than the WMCA treatment group. This may have led to the health impacts and improved condition management seen in SCR.
• Differences in the two labour markets were assessed using labour market information but this could not be linked to trial data. Nonetheless, it showed SCR to have a buoyant labour market whereas the labour market in WMCA was weaker. It suggested that the trial’s IPS service had a greater effect in this context although wider evidence on IPS would not support this point.
• The analyses found that compositional differences did not account for the differences in outcomes; there was no effect from the COVID-19 pandemic on the outcomes of recruits; and the travel incentive introduced in SCR had not changed perceptions of transport as a barrier so was unlikely to account for differences.

While the rate of employment outcomes from the 2 OOW IPS services was broadly similar (at 25% for SCR OOW and 22% for WMCA), the employment outcomes for the 2 OOW control groups were markedly different at 27% in SCR and 18% in WMCA. A greater proportion of the SCR OOW control group saw employment outcomes than the SCR OOW treatment group, and employment outcomes for the SCR OOW control group were 9ppt higher than for the WMCA control group.

A finding from the impact analysis is pertinent - that employment outcomes for the SCR OOW control group by the 12-month stage recovered to the pre-randomisation level whereas in WMCA they did not. In considering what might be driving the higher level of employment outcomes for the SCR OOW control group, it is necessary to consider business as usual experience as well as labour market factors.

3.4.1. Experiences of business as usual compared to treatment

Table 6.5 in the Appendix shows the sources of wider support that were drawn on based on the 12-month survey for the 2 OOW trial groups and showing treatment and control group differences. This shows:

• on employment support; that both control groups were more likely to use Jobcentre Plus services than the treatment groups, and there was greater take up of employment support overall (that is by both the treatment and control groups) in WMCA than in SCR. As noted above, use of wider employment services does not appear instrumental to the employment outcome.
• on health; both control groups accessed health support to a similar degree (44% SCR OOW control group and 42% WMCA control group); however the SCR treatment group was far more likely to access health support (50%) than the WMCA treatment group (39%). As above, improved health and condition management increased capability and capacity to secure work.

This raises a question of whether health support to the SCR OOW control group, in addition to business as usual employment support, led to better employment outcomes. The evidence is weak, as similar rates in the 2 OOW control groups accessed health support and the same rate (63%) reported improvements to health condition management in the 12-month survey.

In contrast, the evidence for a strong health focus in the IPS services in SCR may indicate that employment support was a lesser focus, leading to fewer employment outcomes for the SCR OOW treatment group when compared to the SCR OOW control group. However, it appeared to have a positive effect on enabling people to make best use of health services to improve health outcomes.

3.4.2. Labour market effects

Moving on from support experiences, the analysis of the pandemic and the trials found notable differences between the SCR and WMCA labour markets. The relative buoyancy of the labour market in SCR could have facilitated the ease with which the control group found employment without IPS support – although equally it would be expected that the treatment group could benefit from this. It is possible that a focus on health and wellbeing and managing conditions before seeking employment became a focus, conflicting with IPS principles. However, there is no evidence that can confirm this. The employment rate was higher in SCR than in WMCA across the lifetime of the trials (73% compared to 68% in May 2018). This could indicate greater employability among the SCR population in general or that as the labour market in SCR offers more employment opportunities than WMCA, additional employment support (the IPS service) in WMCA can make a greater difference to outcomes.

This possibility was explored further by examining the difference in the employment outcome after controlling for the influence of observable characteristics in the data. The difference between SCR OOW and WMCA remained substantial. This added weight to the belief that the difference in employment outcomes observed could be driven by different labour markets. It may indicate, more specifically, that there is a greater role for IPS in more depressed labour markets.

Confounding this, however, is substantial and robust evidence emerging from recent systematic reviews and meta-analysis into IPS which find no significant association between the effect of IPS and rate of unemployment in the surrounding labour market (Brinchmann et al., 2020; Metcalfe et al., 2018; Modini et al., 2016). However, a key difference may be that these studies were not conducted during a health pandemic that had significant labour market effects.

3.5. Factors that do not explain the differential impacts

While it was difficult to say with absolute certainty whether and how the above factors affected the impacts that were observed, there were some theories that could be disregarded as there was no evidence for them.

• While there were compositional differences between the trial populations in each site, with those in WMCA harder to help due to greater prevalence of stress, anxiety and depression alongside longer histories of worklessness, the impact analysis showed compositional differences alone were insufficient to explain the difference in impacts between the 2 sites.
• The introduction of a travel pass for unemployed people in SCR who found work for a period of up to 12 weeks during the trial period was explored in light of the difference in treatment and control group employment outcomes and as transport cost was frequently cited as a barrier. Survey evidence showed no change over time over the experience of transport-related barriers for either treatment or control group suggesting that the incentive did not explain differences in the employment outcomes between the trial groups.
• The first national lockdown of the COVID-19 pandemic came into force in March 2020, part-way through the support period for the large, final cohorts entering the trial. Cohort analyses were conducted in the impact analysis and descriptive analysis of the survey. These analyses established there was no evidence of an effect on the impacts observed for the trials.

3.6. Concluding points on explaining the differences in impact between the sites

The above discussion has set out a range of factors that might help explain the differing impacts seen from the trials. It is not and cannot be conclusive. It suggests that there was a differing emphasis on health and wellbeing in SCR, and employment in WMCA. The stronger connection between health and employment systems in SCR might have led to the health impacts observed there.

There were also IPS service differences that might better support improvements to health and wellbeing or conversely employment status. Longer, less frequent meetings might have allowed greater time for discussing health and wellbeing and condition management but may have allowed lapses in the focus on employment. In contrast, more frequent but shorter meetings could have kept up the momentum of job search, offered rapid troubleshooting on CVs, applications forms and so forth, but perhaps offered less time to focus on health and wellbeing support. This perhaps forms the most compelling evidence since it is supported by high quality evidence on the effects of IPS services documented by systematic reviews and meta-analyses (Brinchmann et al., 2020; Metcalfe et al., 2018; Modini et al., 2016).

Alongside these trials-related factors were labour market differences although the IPS literature indicates these would not have an effect. The SCR labour market at the time of the trials was more buoyant with increased employment levels generally and more opportunity than the WMCA. Notably, the SCR OOW control group saw at 12 months a return to pre-randomisation levels of employment whereas the WMCA control group did not. This indicated that business-as-usual performed better than the IPS service in SCR – perhaps because a leaning towards health support in the SCR IPS service which may have distracted from or delayed the focus on employment. From another perspective, it appeared greater additionality was delivered by the employment-focused IPS service in the weaker WMCA labour market that supported and enabled employment outcomes to emerge, despite a wider array of other employment programmes being available.

4. Learning from the trials

4.1. Lessons for the disability employment gap

• The economic case from the trials shows that addressing health condition management and achieving health referrals alongside employment generates returns to individuals and the economy.
• The strengths-based approach brought by IPS meant the treatment groups felt well supported. They appreciated focusing on job roles well matched to their goals and capabilities.
• A close connection with employers, and taking a holistic view to obstacles to the labour market, could help increase employment outcomes. The trials’ results indicate increasing the focus on identifying hidden vacancies and job development would be productive.
• There is some evidence for targeting IPS services at individuals in work and for those whose health is poorer, or conditions more chronic. There is also evidence that people out of work with LTHC and higher levels of qualification would benefit more from more hidden vacancies than advertised vacancies.
• The evidence suggests the need for greater differentiation, and exploration of intersectionality in the design of employment support programmes, to ensure that people from all backgrounds find the support suitable to their needs.

In the years up to the start of the pandemic the general trend in disabled people’s employment had been positive, with an increase in the number (4.4 million) and rate of employment (53.5%). Narrowing the gap between the employment rates of disabled and non-disabled people (known as the disability employment gap) would improve outcomes for disabled people. The government set a goal of 1 million more disabled people in work between 2017 and 2027 (DWP and DHSC, 2017). While the pandemic initially reversed prior positive trends, and the disability employment gap grew by 1 ppt, it began to narrow again from early 2021 (ONS, 2022).

There are also increasing numbers of working-age people reporting long-term health conditions (LTHC); the figure rose by nearly 2 million (17%) between 2013 and 2014 and between 2020 and 2021, largely driven by increased reporting of mental health conditions (ONS, 2022). The pandemic has also had an effect: labour market information from June 2022 showed economic inactivity due to LTHC was at its highest for 20 years (Wilson, 2022). Additionally, there are differences in the prevalence of disability and LTHC in populations between places; illustrating this, a higher proportion of the working age population are disabled in SCR (25%) than in WMCA (20%) (ONS, 2022).

The trials offer insight into support that might help address the disability employment gap from 2 perspectives: supporting people with LTHC into work; and supporting working disabled people and working people with LTHC to remain in work.

4.1.1. Features of IPS that might help to address the disability employment gap

With health identified as the most frequent barrier to work by recruits, and the net economic returns from the IPS services primarily resulting from health outcomes, this suggests jointly considering health and employment is important in the design of employment interventions to support disabled people and those with LTHC. Around 67% of all treatment group respondents felt the IPS support had been helpful to their ability to manage their health conditions. A majority found the IPS services helpful in referring them to other health support, although this differed between trial groups and may be linked to process evaluation findings on the variable extent of co-location and collaboration with health partners found between sites.

The treatment group valued a strengths-based approach to engagement and felt well-supported. The majority of treatment group respondents had positive views of the IPS support, and where they were not positive, views were neutral rather than negative. In the qualitative research, claimants in the treatment group drew contrasts between Jobcentre Plus services and their IPS service, highlighting preferences for IPS approach which focused on finding an appropriate job. A strong majority of treatment group respondents said their employment specialist understood their needs ‘a lot’, with qualitative data suggesting that rapport with the employment specialist was important to the helpfulness of support; this is common in ALMP.

Based on trial findings, it is necessary for employment interventions to help mitigate systemic barriers to work by: working with employers, using discretionary funding, and taking a holistic approach. Views about barriers to work between baseline and the final survey changed little, despite respondents feeling more motivated to work and better able to manage their job search. Recruits (treatment and control) in the OOW trial groups were more likely to say the cost or availability of transport was a barrier over time.

Many barriers to work, such as transport, were systemic meaning employment specialists would not be able to fully resolve them. While they might improve how people manage problems related to transport availability (such as whether they have a driving licence) or caring responsibilities, the barrier itself is unlikely to removed completely using an IPS style intervention. While any employment intervention would be similarly challenged to tackle barriers that are dependent on the availability of other schemes and sources of funding, working directly with employers might offer opportunities, particularly in a tight labour market. This could involve discussing flexible working to overcome childcare barriers and employer-funded solutions to transport barriers.

The CMO analysis found that the support received by the treatment group did not consistently help them overcome contextual factors, and recruits continued to face constraints to work. Shortcomings in the trials’ support mechanisms such as a reliance on general job search rather than accessing hidden vacancies, generic rather than tailored support particularly for more highly skilled participants, and not discussing possible workplace adjustments with employers, were likely to have contributed to these systemic barriers to work remaining for some in the treatment group. The intersections with personal context also suggest individuals may need other forms of support alongside IPS type services.

4.1.2. Which groups might benefit most from IPS?

The impact evaluation explored the extent of variation in impact between the treatment and control groups for some subgroups by gender; age; work experience in the 2 years prior to randomisation; severity of health problem at randomisation (as captured by the EQ5D5L variable); and cohort to reflect the potentially varied effects of the pandemic on outcomes. There were no differences in outcomes for the gender, work experience and cohort subgroups; that is, these observable characteristics were not linked to differences in any outcomes.

However, the impact analysis for the SCR IW group found a significant variation in the impact on health depending on an individual’s health at randomisation, with stronger health impacts seen for those with poorer initial health. This trend was not observed for either of the OOW groups. This lack of subgroup variation indicated that targeting similar interventions for those out of work on specific groups, for example, based on duration out of work, or severity of health condition would be unlikely to result in an increase in outcomes. However, there might be a case for targeting this more intensive support on those with poorer health who remain in employment.

Further analysis of the evaluation data could be undertaken to explore any difference in outcomes by prior level of qualification. The final survey found that respondents with a degree-level qualification were the least likely to find the support helpful in managing health conditions and making adaptations to the workplace compared to those with other levels of education. This might be explained by the approach to sourcing vacancies in the IPS services (see the Implementation and 4-month outcome report). In contrast with IPS principles which stress regular face-to-face engagement with employers to source vacancies, the IPS services focused heavily on advertised vacancies on commonly used jobs websites, which tended towards entry level and low skilled work less well matched to the goals of those with higher qualification levels. On this, the literature on types of work gained from IPS indicates a predominance of entry level and lower skilled roles, elementary occupations and minimum wage jobs, although this may reflect the populations receiving the services having higher needs than HLT recruits (see Heslin et al, 2011; Howard et al, 2010; Latimer et al, 2006; Lones et al 2017; and Oshima et al, 2014).

Perhaps understandably in the trials, those in the treatment groups with higher levels of qualification in the qualitative research felt support could be more tailored to their qualification level although this was not a factor explored in the subgroup analysis of impact. This is a common theme in the ALMP literature. The combined evidence suggests the need for more differentiation, and exploration of intersectionality in the design of employment support programmes, to ensure that people from all backgrounds are as likely to find the support suitable to their needs.

4.2. Lessons for quality of work

• There was limited evidence of the trial having an impact on job quality indicators where the treatment group had moved into work.
• The strongest signs were found for the SCR IW treatment group who showed higher job satisfaction than the SCR IW control group in the final survey.
• Qualitative evidence suggested that the degree of personalisation in the support, and effort to support job carving and job design, were effective in increasing perceptions of job quality.

What good quality work means varies between people, as individuals value different things depending on their circumstances and preferences. Measures include earnings, as well as workers’ perceptions of their jobs.

The impact evaluation found no significant impact on earnings for any trial group. The final survey explored perceptual measures amongst recruits who were working including the influence they had over: the tasks they do; the pace at which they work; how they do their work; the order in which they carry out tasks; and the time they start or finish their working day. No significant differences were found between trial groups. However, the 12-month survey showed that the SCR OOW control group felt their employment was more precarious than the SCR OOW treatment group, feeling, for example, that work makes it harder to manage their health (44% compared with 27% in the treatment group). This focus should not be lost for future delivery if the therapeutic value of work is to be tested.

Overall, recruits who found work after a period of unemployment tended to be more satisfied with work than those who had been in work at the start of the trial (that is, the SCR IW trial group). This difference in job satisfaction might simply be explained by the varying levels of job satisfaction found among employees by length of service; with newer employees tending to be more satisfied at work then employees with longer service history (see for example Dobrow and Ganzach, 2014).

There were differences in perceptions of job satisfaction and work quality between the SCR IW treatment and control groups, indicating the trial could have positively influenced job quality and aspects of work for this group. Respondents in the SCR IW treatment group (56%) were more likely to feel ‘quite satisfied’ or ‘very satisfied’ with their current job compared with those in the SCR IW control group (47%). Also, the SCR IW control group (18%) was more likely to say their current job matched their skills and interests ‘not at all’, compared with the SCR IW treatment group (12%).

The qualitative evidence indicated that employment specialists were able to support the treatment group to understand the working conditions that would best suit and support them to manage their health conditions at work, with a focus on job sustainment, rather than job satisfaction. Some discussed the importance of job carving to help make the experience of work more manageable in relation to health. An interviewee from the SCR IW treatment group described how their employment specialist had helped them negotiate their return to work through agreeing reduced management responsibilities. Another described how discussions with their employment specialist meant they had taken on additional management responsibilities at work.

4.3. Lessons for intervention design

• For disabled people and those with LTHC who are out of work the trials suggest bringing together roles supporting individuals with those supporting employers, and that directly involving employers could increase capabilities around job (re)design and adaptation to support transitions into the labour market.
• Ensuring health support is being accessed and discussing this during employment support meetings could help condition management, and build capacity to re-engage with the labour market.
• Locating employment services in health settings could reach people currently not receiving support.
• Disabled people and those with LTHC who are in work but struggling have differing orientations. Some want support to improve the existing job whereas others want to explore options to change jobs. Vocational profiling can help.
• This is needed in combination with providing support to employers on adjustments and job design.
• In turn, this can increase managers’ and the organisation’s capability and willingness to recruit and retain people with LTHC.

4.3.1. Lessons for the design of employment support for people with health conditions who are out of work

Some elements of IPS such as rapid job search, a focus on the incentives of employment, and personalised support from the same advisor across a period of time, are features of ALMP that are known to be effective. Other elements such as case conferencing, no exclusion criteria, and a requirement for 6 employer engagements per month make IPS stand apart.

Within Jobcentre Plus and contracted employment services, there is a tendency to separate employer-facing roles from those supporting individuals. In contrast, in IPS, the employment specialist coaches the individual and liaises with employers. In business as usual support, case conferencing between advisers supporting people with health conditions, advisers working with employer and employers themselves might elicit increased understanding of the adaptations and flexibilities required to support people with health conditions into work.

The links between health and employment services in SCR enabled recruits to better manage their conditions and contributed to positive health outcomes which delivered economic benefit. There could be scope for increased integration between health and employment systems, particularly with regards to mental health support.

However, across both trials, it is also notable that many recruits were not in contact with Jobcentre Plus, which suggests a need for wider outreach, through health settings, or community settings. Providing access to joined up health and employment support in these spaces might prove effective. There could be a role for Combined Authorities or Local Authorities in understanding the most appropriate settings in each area.

4.3.2. Lessons for the design of support for working people with health conditions

Lessons can also be drawn from the trials for the design of support for working people with LTHC. Trial evidence indicated a positive effect both on the sustained employment measure and the health and wellbeing outcomes for the SCR IW trial group (at the 90% confidence level). Other studies also indicate a positive effect of IPS for IW groups (Li-Tsang et al., 2008; Ferguson et al., 2012; Marshall et al., 2014; van Veggel et al., 2015; Lexen & Bejerholm, 2016; Nygren et al., 2016; Reme et al., 2015, 2019). A mix of individually-centred and employer-centred activities is required.

The SCR IW group differed from OOW recruits in several ways. The surveys showed they were less likely to report other work-related barriers (transport, difficulty finding a suitable job, a lack of experience). They had higher levels of education and were more likely to own their own homes, and they also had longer work histories pre-randomisation. However, they were similar in respect of health; all respondent groups reported that health was their primary barrier to employment.

The SCR IW group had differing motivations for joining the trial. Some wanted support in managing difficult situations in their current employment, whereas others sought help to understand their options for changing jobs. This indicated a need for different sorts of conversation and support than when engaging people who are OOW, but vocational profiling still offers a useful tool.

Amongst respondents who were working at the time of the final survey, satisfaction with the current job varied significantly between the trial groups. Just over half (52%) of respondents in the SCR IW group were either ‘quite satisfied’ or ‘very satisfied’ with their current job. This was lower than the SCR OOW (74%) and WMCA (73%) group respondents. It is unclear whether this was because they remained in the same job and had not been able to achieve new adjustments, or had moved jobs but still found their situation unsatisfactory. It might simply be explained by the varying levels of job satisfaction found among employees by length of service; newer employees tend to be more satisfied at work then employees with longer service history (Dobrow and Ganzach, 2014). Nonetheless, it indicates remaining underlying issues with regards to job satisfaction for many of the SCR IW group, possibly suggesting the employer-employee relationship was not as good as it could be.

The trials’ Theories of Change (ToC) intended that work with employers was central to creating change. Alongside individual conversations, in SCR, the trial sought to engage employers through workshops and information sessions about health in the workplace. Employers who attended and took part in interviews said it made them personally more confident in supporting ill or disabled employees, although this did not necessarily result in company-wide change. The CMO analysis indicated that organisational change may happen slowly over time. For example, an SCR employer found that the process of making adjustments improved their own and organisational awareness of how to support employees with LTHC. In turn, this made them more willing and confident to employ people with LTHC in future. Other changes in employer attitudes and behaviours included a greater focus on workforce wellbeing, training sessions on mental health, and initiatives such as health and wellbeing weeks. Combined, this evidence shows ongoing support for employers can help build confidence to support people with LTHC into work and create a supportive working culture.

4.4. Lessons for IPS delivery

Some key lessons that can be derived from the implementation research that might inform future service delivery include:

• a need for regular training particularly on engaging with employers for employment specialists with limited prior experience of doing this
• ongoing effort to maintain the engagement of GPs and health partners, and to build a shared agenda on the value of work for people with LTHC
• locating employment services with health services to support integrated working and to reach people who are not engaged with the employment system
• adhering to IPS principles on employer engagement to surface hidden vacancies and build capability and confidence for employing people with LTHC
• understanding employers’ needs and experience; employment specialists found it easiest to engage with those with some experience of employing people with LTHC, and those who were struggling to fill vacancies
• caseload level and mix is important to quality; lower caseloads focused either on OOW or IW enabled more time for employer engagement

4.4.1. Need for ongoing staff training

Employment specialists are expected to take on several different roles and tasks as part of their job, and training is vital. The implementation and 4-month outcome analysis showed that employment specialists who joined at the start of the trials received extensive training, but this had not been topped up. Newer staff tended to feel they had not had as much training. Employment specialists who had not delivered IPS previously said it was challenging to familiarise themselves with the expected ways of working, particularly on employer engagement. They noted that effective delivery of IPS requires expertise in health and wellbeing, and in employment support. Many came to the trials with experience without both specialisms, except for those who had delivered IPS before, and so required support in the additional specialism. A common request was for training on employer engagement, which for many was challenging. For services operating over a long time period, regular refresher training would ensure effective practices and emerging lessons for support are captured and shared across teams.

4.4.2. Improving GP and health partner engagement

The IPS manual envisages employment specialists and health practitioners working as a team to assist people with their employment and education goals, providing consistent messages and support to help them achieve those goals (Becker et al., 2019). The trials tested how this would work in primary and community settings. They showed that engaging with GPs and health partners was a key delivery challenge. Despite early engagement, it was difficult to secure sustained involvement. Nonetheless some effective practices emerged.

In SCR, targeted webpages, videos, and case studies of the treatment group were useful resources for gaining GP support, as were regular contact and co-location. Approaches which were sensitive to the lack of time GPs and health professionals experience, such as sending text messages to patient lists and stapling trial leaflets to Fit Notes, improved referral rates. In WMCA, the GP engagement strategy included targets and incentives for referrals, alongside increased communication and sharing feedback. Some co-location in GP surgeries took root and was effective. Notably, the health sector provider found it relatively easier than other providers to make these arrangements, due to already being part of the NHS brand. However, across sites there was limited evidence of a shared agenda to support people into employment. While the trials achieved links between employment and health systems, they did not achieve the integration envisaged by IPS. Continuing work on this should help this progress.

4.4.3. Co-location has multiple benefits

The IPS services were delivered flexibly, in various locations based on individuals’ circumstances and preferences, and this was positively received by the treatment groups. For some, it ensured they got out of the house and into the community; others could engage in a situation that was comfortable to them. Being flexible on location helped employment specialists keep those with complex needs engaged in the trials by being sensitive to their needs. Where co-location and integration with other services had been possible, the treatment group received better support, with a degree of joined up care, cross-referral, and signposting, if not full case conferencing as envisaged by the IPS fidelity scale.

4.4.4. Follow IPS principles on employer engagement

Employer engagement is a core part of IPS, and vital for accessing hidden jobs and supporting people into work, including advising on job carving. It is commonly seen as a challenge within IPS services (Picken et al., 2021; Fyhn et al., 2020). In both trial sites, employment specialists believed that engaging with employers was difficult. This was most prominent amongst staff with limited experience of engaging with employers, often those with health rather than employment backgrounds. IPS stakeholders stressed the importance of employer engagement, which builds trust and relationships over time, and focuses on enhancing employer capability to recruit disabled people and redesign work to be more accommodating of the requirements of people with health conditions.

The format for employer engagement changed in the pandemic to using phone and video conferencing which had some benefits in helping employment specialists manage the demands of their role. Some stakeholders believed that changes to the expectations in the IPS for face-to-face engagement may be helpful although retaining an element of this was important.

4.4.5. Effective ways to engage employers

Employers who engaged directly with the trials typically already had experience of hiring people with health conditions. They tended to offer flexible hours, had experience supporting people back into work, or had targets for supporting disabled people. It was more challenging to work with employers without this prior experience. Employers who had difficulty recruiting or who were recruiting regularly were also more likely to engage with the IPS services. In future delivery of IPS in contexts similar to HLT it may therefore be beneficial to further consider different employer groups and their motivations for engaging at the design stage.

It was noted that a key barrier when recruiting people with health conditions was employer understanding of the cost of reasonable adjustments and how the government could support these. General information sharing was therefore a potentially important way to overcome employers’ concerns about supporting people with health conditions into work. In SCR, employer workshops, on themes such as supporting employees with mental health conditions, were also felt to be a useful way to start conversations about workplace health with employers.

4.4.6. Caseloads and time management need flexibility

Effective IPS support requires a caseload that enables employment specialists to manage the multiple aspects of the role, including employer engagement, generating referrals, and supporting individuals. Low caseloads are a feature of IPS – around 25 people. The service design for the HLT allowed for caseloads to be up to 35. Employability specialists reported mixed experiences of caseload, and with the recruitment flows and peak at the end of the recruitment window, caseloads fluctuated over time. Where they were at the higher end of the spectrum (around 30), some reported challenges in implementing IPS due to the lack of time to lead this in full for all in their caseload. Additionally, managing both IW and OOW trial groups, which was a unique feature of the trial in SCR, meant some staff were working additional hours, evenings, and weekends to accommodate the different preferences for meeting times between these groups. This latter point particularly suggests the need for careful consideration of the differing needs, as well as preferences for timing of support, of people who join the service from a position of employment compared with those who are out of work. Allowing employment specialists a single focus either on working people or those who are not working at the point of joining the service may help more personalised approaches to emerge and lead to improvements in the job design of employment specialists.

5. Appendices

5.1. Design and implementation of the trials

Figure 5.1 illustrates the organisations involved in the trial design and implementation, and the links between them.

Figure 5.1: Organisations involved in design and implementation

WHU was the sponsor and took a holistic role in terms of both overseeing the work of the evaluation consortium, and supporting the 2 local sites via site leads and a Trials Integrity Manager. WHU was also the data controller for the evaluation and trial data; it commissioned the Office for National Statistics (ONS) to be the data safe haven for the trial.

The Institute for Employment Studies (IES) led the evaluation consortium and Professor Stephen Bevan from IES held the role of Chief Investigator.

Sheffield CCG was the data controller for SCR trial and Wolverhampton CCG was the data controller for the WMCA trial.

5.1.1. Designing and commissioning the trial

In WMCA, the trial was part of the Thrive West Midlands action plan developed by the West Midlands Mental Health Commission in 2016, although the focus of the trial was broader than mental health. The trial was championed by the West Midlands Combined Authority (WMCA) which formed in June 2016, with procurement of the trial led by Wolverhampton CCG in conjunction with Arden & GEM Commissioning Support Unit (CSU). The design and set-up for delivery of the trial was led by Social Finance who maintained the contract management and payment of providers during the recruitment and support phase. Service mobilisation and delivery was supported by the Thrive into Work central programme team, based at Wolverhampton CCG. The trial was contracted in lots, which mapped to specific geographic areas in the sub-regions. The providers selected were: Remploy; Prospects and their supply chain which included Enable (an organisation that supports people with mental health issues), Better Pathways (a mental health charity) and Health Exchange (an organisation that aims to work with people to achieve better health and wellbeing in communities); and Dudley and Walsall NHS Trust. The trial was governed by the Health and Wellbeing Board of the Combined Authority.

SCR has a commitment to inclusive growth and enabling all residents to benefit from the employment opportunities in the city region, and chose to design a trial to support both in- and out-of-work residents. A group was established to lead the trial in SCR consisting of the SCR co-design lead and health colleagues from each of the 5 areas; for example, Directors of Public Health, Directors of Health Improvement, and CCG Director of Quality and Care, alongside representation from the WHU. SCR appointed a dedicated health engagement role to lead on stakeholder engagement, to inform the design of the trial. This resulted in plans to use GP surgeries and well-used community venues for delivery. There was also engagement with representative bodies for the voluntary and community sector, and directly with third sector organisations, to gain their support to signpost potential recruits to the trial. Alongside this, these organisations were also consulted on a self-referral process for recruits. Additionally, there was engagement with Occupational Health Partners who inputted into the design of the in-work support offer.

The trial in SCR was performance-managed by city region staff and its commissioning was led by Sheffield CCG on behalf of the 5 CCG areas involved. South Yorkshire Housing Association (SYHA) successfully bid to deliver the contract. SYHA had previous experience of delivering IPS through a contract for Building Better Opportunities. ^{[footnote 5]} The SCR Skills, Employment and Education Board was the governance body with lead responsibility for trial performance management, feeding into the overall Combined Authority Board. A regular steering group was established within SCR, bringing together SCR and local authority partners with health partners from the Sustainability and Transformation Partnership (STP) and CCGs, as well as national partners, to review the progress.

It is worth noting that around summer 2019, the SCR Mayor introduced a travel pass for a period of up to 12 weeks for people transitioning from reliance on benefits (where applicable) to a position of reliable income, that is employment. This support was offered across the employment schemes in SCR including the Working Win service although level of take up within the trial population is not known.

5.1.2. Eligibility criteria for the trials

The trials were designed to empirically test whether IPS-LITE for people with mild-to-moderate mental or physical health conditions had an impact in improving health, wellbeing, and employment outcomes beyond what would be achieved with ‘business as usual’ (BAU).

In SCR, the trial was open to 2 groups:

• individuals with a self-defined low/moderate mental health and/or physical health condition which is an obstacle to their desire to be in sustained employment
• individuals in employment but who were either off sick or struggling in the workplace due to a self-defined low/moderate mental health and/or physical health condition

In WMCA the trial was open to:

• people who had been out of work for more than 4 weeks, who wanted to find work, and who were disabled or had a health condition which presented an obstacle to them gaining work

Additionally, trial recruits needed to be able to give informed agreement to participate in the trial, be aged over 18, want to take part in voluntary support to find and sustain employment and not be part of other employment interventions beyond Jobcentre Plus employment support from work coaches. Furthermore, recruits could not be identified as having severe health conditions which would be indicated by them being on a Care Programme Approach, receiving Community Mental Health Team (CMHT) support or identified on a Serious Mental Illness (SMI) register.

5.1.3. Overview of evaluation design

The evaluation was intrinsically linked to the design for IPS service delivery, and both the evaluation and the service delivery plans were submitted for approval by the Health Research Authority (HRA). The national evaluation was built around a randomised controlled trial (RCT) methodology. Using this, recruits who agreed to take part were randomly allocated to either a treatment group (the new IPS service) or a control group (BAU). The evaluation approach consisted of a process evaluation, an impact evaluation, and an economic evaluation.

An overview of the evaluation methodology is shown in Figure 5.2 and discussed in chapter 6.

Figure 5.2: Overview of evaluation

5.1.4. Ethics, HRA, registration and local approvals

An important part of the design phase was to ensure the trials were conducted ethically, with regard to the safety and wellbeing of recruits and researchers, and to obtain the necessary permissions from research governance and ethics boards. Because the trials took place in the health system, it was necessary to apply for approval from the Health Research Authority (HRA). This involved completing an Integrated Research Application System (IRAS) form, submitting this to a Research Ethics Committee (REC), and providing details about the trusts and CCGs that would be involved, and what their role would be.

The evaluation consortium led on identifying potential ethical issues and/or threats to the safety and wellbeing of recruits and researchers proposing mitigation measures. These were shared with the sites for agreement prior to inclusion in the IRAS form. Key issues were: ensuring informed and voluntary agreement to take part; setting out measures to protect recruits’ confidentiality; agreeing a disclosure policy (should the research team learn about risks of harm or potential serious criminality during the course of research); and having a system for how adverse impacts on recruits would be monitored.

Having agreed on a proposed approach to the ethical issues, the evaluation consortium designed key documents and materials to be used in the trial:

• participant Information Sheets (PIS; in accessible and plain English versions)
• agreement forms (in accessible and plain English versions) *opt-out forms
• consent and opt-out materials for other parts of the research (including surveys, interviews and observations)

Given the importance of the patient-facing materials, the design process involved input from multiple stakeholders: initially from the trial sites, WHU and consortium members, and then wider stakeholder groups. The process also involved consultation with patient groups who also tested the survey instrument.

The patient groups were engaged in the development work through user testing workshops carried out in each site in June 2017. The aim was to gather feedback on the trials’ consent and information materials. The patient groups from each site comprised adults with lived experience of mental health problems (mild to severe), substance misuse and physical disabilities, drawn from established patient stakeholder groups in each site. Those taking part were recruited by members of the delivery team in each site. The staged consultation process, with input from data owners following the user testing events, meant it was not always possible to adopt user recommendations. For example, feedback from users was that the PIS and agreement forms were too long; however, data owners required that additional detail be built into the documents in order to meet legal requirements.

The trials were considered by the HRA Research Ethics Committee (REC) on 19 September 2017. Following a period for clarifications, Health Research Authority (HRA) approval was issued on 1 November 2017.

Following this, the evaluation consortium proceeded to request local Capability and Capacity (C&C) checks in each of the 16 trusts and the CCGs in which the trials took place, working closely with those involved in implementation in SCR, WMCA and the Clinical Research Networks in both sites. These are a necessary procedure for HRA-approved research to underpin the involvement of health organisations.^{[footnote 6]} The pace with which these checks and permissions were gained varied between and within the trial sites. In WMCA, 5 trusts had granted C&C checks by early August 2018, with another following in October 2018 (Birmingham Women’s and Children’s NHS Foundation Trust), and 2 others not until early 2019 (The Birmingham and Solihull Mental Health NHS Foundation and Royal Wolverhamption NHS Trust). In SCR, in contrast, all trusts had completed and been granted C&C checks by all Trusts by May 2018 when delivery started (in the main body of the report).

During delivery it was necessary to amend the HRA approval given for the trials at various points, for example, for the development of new materials. These amendments included the use of text messages sent by GPs and local radio adverts in SCR.

5.1.5. Data management

WHU was the data controller for the trial data held by the evaluation consortium and ONS, and the data was processed on the legal basis of public task. WHU led on the design of the data architecture and data flows. The Office for National Statistics (ONS) was appointed by WHU to be the data safe haven with data from the delivery and evaluation consortium transferred into and out from the ONS as required.

The agreement process sought permission from recruits for their personal information to be collected during the trials and retained for up to 3 years after they ended, with personal information used for service delivery, and shared and linked to other information held by government departments and NHS Digital for research purposes. Recruits also agreed that after 3 years, an anonymised version of their information could be stored in the UK Data Archive to support future research.

Recruits had the right to withdraw from the trials and stop receiving the service prior to the planned end. Where recruits requested to withdraw from the trial, the information already provided up to that point continued to be used for research and analysis and could be linked to administrative data, but these recruits were not asked to participate in any further primary research, such as surveys or interviews. At the point at which the data were drawn by the end of the evaluation, 287 recruits had withdrawn from the trials, and 10 had died.

Under the General Data Protection Regulations (GDPR), in some circumstances, people can ask for an organisation that holds data about them to delete it. This is known as the right to erasure (sometimes called the ‘right to be forgotten’). However, the right to erasure is not absolute, and does not apply if the processing is necessary to meet certain conditions. For the HLTs the right to erasure did not apply because processing was necessary for the performance of a task carried out in the exercise of official authority. Also, where data are used for research or statistical purposes, erasure is likely to render impossible or seriously impair the achievement of that processing. However, there were some requests to erase data which were upheld, where an individual provided a strong case in favour of the erasure. In total, 5 instances of full data erasure were granted. These cases are not included in any reporting of the trials.

At the start of every primary data collection point, recruits were asked to consent to take part in the specific research activity and given details about how their data would be used.

These protocols meant that there were different numbers of recruits eligible to take part for each batch of 2 surveys and available to the qualitative research, compared with those found in the baseline data collected at randomisation. The 4- and 12-month surveys were undertaken on a rolling basis and therefore the number of individuals in each batch could alter where there were cases of withdrawal or erasure between survey waves. Also, recruits could withdraw in periods between randomisation and the 4-month survey, and between the 4-month and 12-month surveys.

5.1.6. The randomisation tool

Randomisation was carried out by employment specialists during an initial meeting, using an adapted version of a bespoke tool developed by the Behavioural Insights Team^{[footnote 7]} for an earlier trial. The tool was accessed via a secure website. The randomisation process screened individuals for eligibility; requested and recorded agreement to take part in the trial; collected personal data, national identifiers, background information and pre-trial responses to questions covering health and wellbeing and job search self-efficacy using a baseline data collection survey; and finally randomised recruits either to the IPS support group (the treatment group) or to the BAU control group.

Individuals were randomised according to a pre-specified algorithm. This used a permuted-block design. The randomisation ratio (the proportion allocated to the treatment group) could have been allowed to vary across subsamples within the trial populations and over the course of the trials, but this feature was not required; it was the means of controlling the flow into the trials and ensuring staff capacity to deliver IPS was not exceeded. However, given the lower than planned overall recruitment, this was not required. This feature was also set in place so that the trial would accommodate the desire to achieve a 70/30 unemployed/employed split in the SCR trial population. However, again, as recruitment was lower than planned, and using this feature would have reduced the flow of working recruits to the trial – thereby leading to fewer recruits overall, again it was not used.

5.1.7. Training about the RCT

The evaluation consortium ran a training session in each trial site in winter 2017 and provided written guidance, accompanied by videos, to prepare trial staff for key aspects of the trial. Operating as a randomised controlled trial brought responsibilities that standard service delivery would not have. It was crucial that IPS employment specialists followed the protocols and guidance as these aimed to ensure ethical ways of engaging with recruits, and to protect the integrity of the trials and, in turn, the reliability of the results. The training and guidance materials covered: an overview of the evaluation including of the RCT and implications of this; gaining informed agreement to participate from recruits; baseline data collection; the randomisation tool; and the management information requirement.

When leading the initial meetings with potential recruits, employment specialists had responsibilities to:

• Assess eligibility. The randomisation tool contained a set of questions to assess eligibility tailored to each site (see section 5.1.2) and staff had a responsibility to ensure that only people that met the criteria were offered the trials and asked for agreement to take part.
• Give information. Participant Information Sheets (PIS) provided information to potential recruits about the trial. It was intended that these were explained as part of the initial meetings to ensure that people could make informed decisions on the same basis. The accessible and plain English formats PIS were user tested. Both versions were worded to ensure comprehension with the accessible version focused on being suited to those with literacy difficulties.
• Gain informed agreement to take part. Having established interest and eligibility, employment specialists were required to take people through each statement in the agreement form, to ensure they fully understood the implications of each, so that they could make an informed decision about whether or not to take part.
• Collect baseline information. This covered a range of personal information, national identifiers and pre-trial perspectives on health and wellbeing and job search self-efficacy.
• Deliver the randomisation result. Help recruits understand the result and communicate it in a neutral manner.
• Provide information about the next steps. Those in the treatment group were told about making an appointment for their next session and those in the control group were given an information leaflet to enable them to approach other local services for support should they wish.

Agreement was reached that the training sessions offered by the evaluation consortium would be supplemented with locally run training programmes as new staff joined the trial. These necessarily differed between providers. The materials developed by the evaluation consortium were made available to all staff on a dedicated extranet for each trial site, alongside key contact details for the consortium, and frequently asked questions (FAQs).

The steps for engaging and recruiting people are summarised in Figure 5.3.

Figure 5.3: Overview of the initial meeting and randomisation process

6. About the evaluation

6.1. Evaluation approach

The evaluation drew on 3 strands of research using 4 main methods. The process evaluation used 3 waves of qualitative data collection, and multiple reviews of management information (MI). Alongside this were surveys with recruits to the treatment and control group at 3 time points: pre-randomisation (baseline); 4 months after randomisation (4-month); and 12 months after randomisation (final). The descriptive analysis of these formed part of the process evaluation. Impact analyses at the 4-month point drew on outcomes recorded by the survey, whereas at the 12-month point survey data were combined with administrative data held by public bodies to obtain the employment outcomes. The economic assessment drew on data from the trial sites and WHU to understand costs, and used the impact analysis and the administrative data to support the identification of financial and economic benefits.

6.2. Methodology

Using the RCT methodology, recruits who agreed to take part in the trial were randomly allocated to either a treatment group (the IPS service) or a control group (BAU). Monitoring was undertaken throughout the recruitment phase to ensure trial arms remained balanced. The evaluation was underpinned by Theories of Change (ToCs)which set out in schematic form the support activities to be delivered, intended outcomes (for users, health professionals and employers) and hypothesised mechanisms through which change would be achieved.

6.2.1. Process evaluation using Theories of Change

The trial ToCs were developed during the design phase in consultation with local stakeholders in May 2017. These guided the sample design, and lines of questioning in the research tools. The ToCs were updated during summer 2019 at 2 workshops. In SCR, the workshop was attended by 24 representatives from SCR, the provider, DWP, local councils, and CCGs. In WMCA, the workshop was attended by 13 representatives from Social Finance, the providers, WMCA and CCGs.

The process study was undertaken alongside service delivery to help explain how and why the intervention worked (or did not work). It aimed to capture information about delivery that could aid replication should the trial establish positive impact on outcomes. The process evaluation necessarily explored whether and how the mechanisms identified in the ToCs operated in practice and which elements of the trial context affected these mechanisms. Interrogating the ToCs of change forms the key analysis of the process data in the CMO report.

A mixed methods approach was used for the process evaluation consisting of: analysis of management information (MI) collected by the sites; and qualitative fieldwork, including with recruits, IPS staff, health professionals, employers and other key partners and stakeholders. Descriptive analysis of the surveys (see section 2.3.2 also provided information for the process evaluation and ToC assessment.

The providers delivering the IPS service collected management information (MI) on the nature, range and intensity of support delivered to participants (the treatment group) This supported a descriptive analysis of IPS delivery and variation across sites; the exploration of linkages between activities and outcomes; and information required for qualitative sampling. This information was also exploited in the impact analysis to understand intensity of treatment. The evaluation consortium specified an indicative, core list of MI fields required for the process study to aim for consistency between the 2 sites and the 4 main providers. The MI included in this report covers the period from May 2018 to April 2020.

Qualitative research, including in-depth interviews with recruits, staff and stakeholders, explored a range of outcomes and perceptions of how any change occurred, including tracing the pathways between support received, intermediate and longer-term outcomes, and wider contextual factors (such as perceived barriers to work and changes in these over time, and non-trial support received). Interviews involving treatment group recruits also explored experiences of engaging with the IPS services and views on which approaches worked well for them and why.

Samples were drawn from the MI. Demographic sample plans included quotas for each area within SCR and WMCA, and targets for gender, age, and ethnicity. In addition, a quota for IW and OOW recruits was set for SCR interviews. The quotas aimed to capture recruits who had been on the trial for different lengths of time and for the treatment group had achieved different types of outcomes (including soft or intermediate outcomes, as well as work and health outcomes).

Interviews with recruits to the treatment group were undertaken at 3 time points. Those with recruits to the control group took place at 2 time points. This enabled exploration of outcome pathways, and how and why these differed. Half of the recruits interviewed at wave 1 were re-contacted for interview at wave 2, and an additional group were interviewed at wave 2 who were re-contacted at wave 3. This enabled the research to explore the interactions between support, outcomes, and contextual factors as they were revealed over time. For the control group interviews, around a third were longitudinal while the remainder were cross-sectional.

Interviews were complemented with in-depth interviews with delivery staff at 3 time points and in-depth interviews with delivery partners and local stakeholders at the start of the trial. Partner and stakeholder interviews – involving local strategic stakeholders, key referral, and other partners supporting delivery, explored views on the trial delivery, contextual influences and expected impacts. A breakdown by type of interviewee is shown in Table 6.1.

Table 6.1: Completed interviews by time and respondent group

Respondent type	SCR interviews	WMCA interviews	Total
Participants: treatment group	48	48	96
Participants: control group	17	18	35
Participants: longitudinal	34	32	66
Staff	34	36	70
Stakeholders and partners	17	15	32
Employers	8	6	14
Total interviews	157	156	313

Source: Evaluation records

In addition, 6 thematic deep dives were undertaken focused on key emergent issues, between December 2019 and November 2020. They focused on:

• employer engagement – 1 focus group with staff in SCR and 1 focus group with staff in WMCA, and document review
• delivery of employment support – 1 focus group with participants in SCR and 6 interviews with participants in WMCA, and document review
• delivery of job development activities – 5 interviews with participants in SCR and 5 interviews with participants in WMCA, and MI analysis
• engagement of primary and community care – a focus group with partners in WMCA and 2 interviews with partners in SCR (small numbers due to COVID-19), and document review
• adapting delivery to the pandemic – discussions with 2-3 employment specialists in each site, and a desk review of information about how the pandemic was affecting the UK population and labour markets
• understanding costs of delivery – discussions with managers in each trial site and data collection template to understand staffing ratios and tasks across the course of trial delivery

The qualitative data were managed and analysed using a framework approach. This involved the identification of key themes either from the data or from pre-existing hypotheses in the ToCs to develop a thematic framework. This framework was used to classify and organise the data from each respondent. The coded data for each theme was reviewed in detail, drawing out the range of experiences across respondents. Emergent patterns were interrogated in relation to key characteristics of interest, such as differences between in- and out-of-work recruits.

6.2.2. Surveys of recruits

The surveys contributed data to the process and impact evaluations. Recruits were surveyed to understand the effects and impacts of the IPS services. Outcomes were collected at 3 time points:

• the baseline assessment completed by trial staff prior to randomisation
• a 4-month survey administered 4 months post-randomisation (covered in the Implementation and 4-months outcomes report)
• a final survey 12 months post-randomisation (included in the survey descriptive report in the final report series)

To inform the process evaluation, the surveys were designed to collect: key intermediate outcomes identified in the ToCs that might indicate progress towards the primary and secondary outcomes, and the treatment groups’ experiences of the IPS services and perceived impacts.

To support the impact evaluation, the surveys were designed to capture health, wellbeing, and employment outcome measures to supplement administrative data from national data owners.

The outcome measures selected for the 4-month and final surveys formed the basis for the baseline assessment developed for administration by trial staff during the initial meeting. Additional questions were added to collect full contact details, information for data linkage (NHS number and NI number), health condition, language and communication needs and socio-demographic information. The questionnaire was programmed into an online survey embedded within the randomisation tool.

The 4-month survey was placed in the field on 17 October 2018 and was administered to the full sample of randomised recruits who had not withdrawn prior to survey fieldwork commencement in any month (N=9,539^{[footnote 8]}), indicating a response rate of 56.6% (N=5,407) based on this sample. The final response rate was calculated based on the 4-month data set which included 9,785 cases and was 55%.

The final survey was conducted 12 months after recruits were randomised into the trial. Fieldwork was conducted in 17 monthly batches between 12 June 2019 and 25 November 2020. Again, it was issued to all recruits who had not opted out of the trial, that is not just those who responded to the 4-month survey (8,945) and achieved a total of 4,087 interviews, a response rate of 46%.

Survey administration

At each survey point, all recruits who had not withdrawn from the trial received a letter which invited them to complete the survey. The survey was completed either face-to-face or using a web-telephone survey mode, although once the COVID-19 pandemic set in, the face-to-face option had to be closed. During the randomisation appointment as part of baseline data collection, recruits could request a face-to-face survey, for reasons including using British Sign Language as their first language, hearing difficulties, anxiety over using the phone, or personal preference. Following the invitation letter, at each survey wave interviewers made 5 approaches to participants, face-to-face or by telephone, to book an interview. Recruits who were contacted 5 times with no response were coded as refusals.

Recruits who had not requested a face-to-face survey completed the interview by web or telephone. Following the invitation letter, they received an email with a link to complete the survey online. They were also called by a telephone interviewer to complete the survey over the phone. Recruits could choose to complete the surveys using both modes if they preferred, such as beginning the survey online and completing it when phoned by an interviewer. Those who did not complete the survey online were called a minimum of 12 times to maximise participation.

Data collected from the survey were weighted for non-response. Further information about the weighting can be found in the Technical Reports for the Implementation and 4-month outcomes report, and the 12-month survey descriptive report.

6.2.3. Impact evaluation

The primary hypothesis the trial aimed to test was whether IPS improves employment, health and wellbeing outcomes beyond what can be achieved with ‘business as usual’ (BAU). Outcomes were captured through administrative data and surveys and impacts were estimated at 2 points:

• 4-month – 4-month impacts on survey outcomes (presented in the Implementation and 4-month outcomes report)
• final – 12-month impacts on administrative data and survey outcomes (presented in the 12-month outcomes impact report)

Impacts were estimated separately for:

• SCR recruits who were not working when randomised (SCR OOW)
• SCR recruits in work but struggling or off sick when randomised (SCR IW)
• SCR recruits in or out of work when randomised (All SCR)
• WMCA recruits, all of whom were out of work when randomised (WMCA / WMCA OOW)
• a pooled group (covering SCR OOW and WMCA) of recruits who were not working when randomised (All OOW)

The trial protocols and statistical analysis plan (see Appendices to the Implementation and 4-month outcomes report, Chapter 7) identify 3 primary outcomes at the 4-month stage (employment, health, and wellbeing) and 4 primary outcomes at the final stage (employment, earnings, health, and wellbeing). Several secondary outcomes are also identified. All outcomes at the 4-month stage were drawn from the 4-month survey whereas the final outcomes drew on the 12-month survey in addition to national administrative data sets.

The evaluation used different measures of employment for the 4-month and final, 12-month stages. At the 4-month stage, the measure used was whether the individual was employed 4 months following randomisation. At the final report stage, the employment and earnings outcomes were linked from tax records so covered more of the trial population than the surveys. The outcome measure used was whether the individual had been in employment for 13 or more weeks in the 12 months following randomisation. The rationale for this was to align more closely with measures for active labour market policies which explore sustained employment rather than short-term outcomes. However, it must be emphasised this measure is more ambitious than those often used typically in IPS studies, which focus on 1 additional day in employment compared with the control group. The health and wellbeing outcomes at both stages draw on surveys.

The full table of primary and secondary outcomes at 4– and 12-months is contained in the appendix of this report (Table 6.3).

6.2.4. Economic evaluation

The economic evaluation sought to establish whether the benefits of the IPS services exceeded the costs of their delivery. It estimated the return on investment from every £1 spent on the IPS services, based on the evidence from the trial. This included a full assessment of where costs were incurred and the value of particular types of benefits, including noting the costs or benefits to which it was not possible to attach a monetary value. The approach to calculating the return on investment from the IPS services was on that set out in the HMT Green Book.^{[footnote 9]}

Analysis of the management information, survey data and linked administrative data was used to determine the likely benefits. These sources were also used to identify costs arising from the IPS services, supplemented by data from other sources, including the process evaluation and publicly available information. The 12-month impact estimates were used to estimate the value of the benefits that the IPS services produce. The full economic assessment for the trials can be found in the 12-month outcomes: economic evaluation report.

6.2.5. Understanding how the pandemic affected the trial

As noted in section 1, the onset of the pandemic, and subsequent COVID-19 containment strategies including lockdowns, occurred during the final year of delivery of the IPS services. The timing meant that the pandemic particularly affected the job placement and support period for the relatively large cohorts recruited in autumn 2019. Some additional elements of research included:

• Dedicating a deep dive to exploring service adaptation during the pandemic. This involved interviews with employment specialists and managers, and examination of the MI and new data captured about how support changed.
• Leading additional work to analyse labour market information and data sets.
• Leading additional interviews with employers located in the 2 sites.
• Leading additional interviews with IPS stakeholders and other IPS services to understand adaptations to delivery more broadly.
• Reviewing evidence and data to be able to document the effect of the pandemic in the UK. This and all analyses above are included in the report on the pandemic effects.
• Leading ‘cohort’ analyses using the survey data and linked final data set to understand any effects. Findings from these analyses are covered in the 12-month survey descriptive and 12-month impact report, respectively.

6.3. About the evaluation data sources

The evaluation draws together quantitative data from several sources, each collected using different methodologies covering different time points.

The final data set (which combined baseline data collection, 4-month or 12-month survey data, with service provider MI at the individual level and administrative data at the 12-month point) contains information on 9,785 recruits who were randomised and not granted erasure of their data (the latter affected 5 additional cases, with deletion granted before the 4-month data set was analysed; these cases were removed from the evaluation data and therefore do not appear in the baseline data).

Data were collected from all recruits using the baseline data collection tool during their initial appointment with an Employment Specialist. The baseline data set initially therefore included all recruits, but for analysis the 5 that subsequently requested and gained agreement that their data be deleted were removed (N=9,785).

The 4-month and final, 12-month surveys were undertaken on a rolling basis around 4 and 12 months after randomisation, with the fieldwork period spanning 18 months for each survey. All trial recruits were invited to participate in the surveys (except those who had withdrawn from the trials in the lead up to each launching, as that meant that they would not be contacted for primary research). However, some recruits when contacted for the surveys chose not to take part, some did not respond after several reminders/calls for the surveys, and contact details for some may have changed since they joined the trial. Where the service provider MI indicated that the contact details of the treatment group had changed, this information was purposely not updated in the samples provided to the evaluation consortium, to avoid introducing bias between the treatment and control groups.

The responses to the 4-month and 12-month surveys were therefore from a subset of all recruits; the overall response rate to the 4-month survey was 55% and for the 12-month survey it was 46%. There were some differences between the response rates of the treatment and control group recruits, as well as between the 2 trial sites (see the Implementation and 4-month outcome, and the 12-month survey descriptive report for more details). Additionally, survey respondents may have chosen not to answer specific questions, or to respond ‘don’t know’, and therefore the base sizes vary between questions. The base size for each question is noted in the tables.

The data set for the 4-month impact analysis used the 4-month survey, linked to the baseline data set, with the service provider MI covering only the treatment group also linked to check compliance. All outcomes at the 4-month point were measured using the survey data. The final evaluation data set comprised the survey of 12-month outcomes that took place between May 2019 and October 2020. The impact analysis used these survey data, alongside national administrative data held by HMRC and DWP. The administrative data were used as the primary source of outcomes for employment, and benefits claiming and off-flow at 12-months. Survey data were the primary source for health and wellbeing outcomes.

Both 4-month and final data sets took account of right to erasure requests up until 23 July 2020. Table 6.2 summarises the scale of each quantitative data set.

Table 6.2: Summary of quantitative data sets

		SCR	SCR	SCR	SCR	WMCA	WMCA	All
		IW	IW	OOW	OOW	OOW	OOW
Data set	Time	T	C	T	C	T	C	Total
Baseline data collection	May 2018 to Oct 2019	1,260	1,259	1,799	1,792	1,837	1,838	9,785
4-month survey: achieved sample	Oct 2018 to March 2020	780	683	1,057	944	1,007	936	5,407
Response rate (%)		61.9	54.2	58.8	52.7	54.8	50.9	55.3
12-month survey: achieved sample	Oct 2019 to March 2021	640	541	762	726	754	664	4087
Response rate		50.8	43.0	42.4	40.5	41.0	36.1	41.8
Service provider MI (treatment group only)	May 2018 to April 2020	1,160	N/A	1,556	N/A	1,772	N/A	4,488

Source: Evaluation data set

The evaluation collected both quantitative and qualitative data with the latter resulting from the range of interviews undertaken for the process evaluation. It is important to note that the mode of data collection affects how that data can be reported. When using quantitative data, it is possible to report on the scale of views held and to enumerate different outcomes. In contrast, qualitative data provides explanatory and illustrative information, but it cannot be enumerated because questions are not asked in a systematic way of a representative sample.

The analysis is structured by considering outcomes for 5 trial groups as follows:

• SCR IW (the group in SCR who were in work at the point of being randomised)
• SCR OOW (the SCR group who were out of work when randomised)
• All SCR (the 2 SCR groups combined)
• WMCA (all of whom were out of work when randomised)
• All OOW (the combination of the 2 out-of-work groups when randomised, that is SCR OOW and WMCA)

6.4. Primary and secondary outcome measures for the trial

Table 6.3: Outcome measures for the trials at 4 and 12 months

Outcome	4-month measure	12-month measure
Primary outcomes
Employment	Measured by the survey Employment: What is your current employment status? If you are currently employed and on sick leave or temporarily reduced hours, please record your normal working pattern prior to taking sick leave or reducing your hours.	Measured by administrative data student (based on HMRC PAYE RTI data)
Earnings	n/a	Measured by administrative data Total earnings in the 12 months following randomisation (based on HMRC PAYE RTI data)
Health	EuroQol-5D-5L (EQ5D5L) from 4-month survey Comprises 5 dimensions: mobility, self-care, usual activities, pain/discomfort and anxiety/depression.	EQ5D5L from 12-month survey
Wellbeing	Short Warwick-Edinburgh Mental Well-being Scale (SWEMWBS). 7-item from 4-month survey.	SWEMWBS from the 12-month survey
Secondary outcomes
Employed	n/a	Employment by month since randomisation (HMRC)
Number of months employed		Number of months employed since randomisation (HMRC)
Earnings		Earnings by month since randomisation (HMRC)
Receiving out-of-work benefits		Receiving out-of-work benefits by month since randomisation (DWP)
Employed and receiving benefits		Employed while receiving out-of-work benefits by month since randomisation (DWP, HMRC)
Working (employed or self-employed)	Emp, as above	Emp, from 12-month survey
Working 16+ hours per week	Emp, as above	Emp, from 12-month survey
Number of weeks working since randomisation	ContEmp2 from 4-month survey During the last 4 months, approximately how many weeks have you been in employment?	ContEmp2 from 12-month survey
Number of continuous weeks working 16+ hours per week since randomisation	ContEmp3 from 4-month survey And out of these [textfill ContEmp2] weeks, how many have you worked more than 16 hours a week (on average)?	ContEmp3 from 12-month survey
Number of continuous weeks working 16+ hours per week	ContEmp4 from the 4-month survey Have these [textfill ContEmp3] weeks been continuous (specifically, without a gap)?	ContEmp4 from the 12-month survey
Job search self-efficacy	Job Search Self-Efficacy Scale (JSSE) from the 4-month survey. 9-item. Self-efficacy relating to finding employment	JSSE from the 12-month survey
Musculoskeletal Health	Musculoskeletal Health Questionnaire (MSK-HQ) from the 4-month survey 2-item.	MSK-HQ from the 12-month survey
Mental health	n/a	General Anxiety Disorder-7 (GAD-7), from the 12-month survey 7-item, and Patient Health Questionnaire-8 (PHQ-8), 8-item on anxiety and depression.
DDA definition health	HCond from the 4-month survey Do you still have 1 or more health conditions or disabilities?	HCond from the 12-month survey
Life satisfaction	Office for National Statistics Personal Well-being Questions from the 4-month survey (ONS-1).	ONS-1 from 12-month survey
Self-efficacy	General Self-Efficacy Scale (GSE Scale) from the 4-month survey 10-item .	GSE from 12-month survey

6.5. Additional tables from the 12-month survey

Table 6.4 Access to additional forms of support among trial recruits

		SCR IW %	SCR OOW %	WMCA %	Total	P-value
Support accessed	Support from Jobcentre Plus	14	39	49	36	0.000
Support accessed	Support through the Work Programme	5	12	16	12
Support accessed	Support from Occupational Health	25	7	8	12	0.000
Support accessed	Support through the Work and Health Programme	6	8	13	9
Support accessed	Support from GP/other primary health services	60	47	40	48	0.000
Support accessed	BITA Pathways Learning & Work service	-	0	4	1
Support accessed	Dudley and Walsall Mental Health Partnership NHS Trust Employment Support Service	-	-	7	3
Support accessed	Black Country Impact	-	-	5	2
Support accessed	Building Better Opportunities	-	-	3	1
Support accessed	Third-sector drug, alcohol or other health services	0	1	1	1
Support accessed	Third-sector employment support services	1	1	1	1
Support accessed	Other	6	10	11	9
Support accessed	None	27	28	25	27	0.125
Unweighted bases		1,174	1,474	1,407	4,055

Base: all final survey respondents who accessed services other than the IPS intervention and who are not currently in employment.

Table 6.5: Access to additional forms of support among trial recruits, and by treatment and control groups

			Trial group - Control %	Trial group - Treatment %	Total	P-value
Support accessed	SCR IW	Support from Jobcentre Plus	16	13	14	0.138
Support accessed	SCR IW	Support through the Work Programme	4	6	5
Support accessed	SCR IW	Support from Occupational Health	23	26	25
Support accessed	SCR IW	Support through the Work and Health Programme	5	6	6
Support accessed	SCR IW	Support from GP/other primary health services	55	64	60	0.003
Support accessed	SCR IW	None	29	25	27
Support accessed	SCR OOW	Support from Jobcentre Plus	42	37	39	0.052
Support accessed	SCR OOW	Support through the Work Programme	14	11	12
Support accessed	SCR OOW	Support from Occupational Health	5	9	7
Support accessed	SCR OOW	Support through the Work and Health Programme	8	7	8
Support accessed	SCR OOW	Support from GP/other primary health services	44	50	47	0.029
Support accessed	SCR OOW	None	28	29	28
Support accessed	WMCA	Support from Jobcentre Plus	54	45	49	0.001
Support accessed	WMCA	Support through the Work Programme	17	15	16
Support accessed	WMCA	Support from Occupational Health	10	7	8
Support accessed	WMCA	Support through the Work and Health Programme	15	10	13
Support accessed	WMCA	Support from GP/other primary health services	42	39	40	0.340
Support accessed	WMCA	None	21	29	25
Support accessed	All	Support from Jobcentre Plus	39	33	36	0.000
Support accessed	All	Support through the Work Programme	12	11	12
Support accessed	All	Support from Occupational Health	11	13	12
Support accessed	All	Support through the Work and Health Programme	10	8	9
Support accessed	All	Support from GP/other primary health services	46	50	48	0.016
Support accessed	All	None	26	28	27
Unweighted base	SCR IW		538	636	1,174
Unweighted base	SCR OOW		717	757	1,474
Unweighted base	WMCA		662	745	1,407
Unweighted base	All		1,917	2,138	4,055

Base: all final survey respondents.

Table 6.6: How much the support received from these services increased or decreased motivation to find employment, by trial group

		SCR IW %	SCR OOW %	WMCA %	Total	P-value
Has the support you have received from these services increased or decreased your motivation to find employment?	Increased a lot	16	22	20	21	0.240
	Increased a little	32	32	30	31
	Decreased a little	7	7	9	8
	Decreased a lot	8	8	8	8
	No effect on motivation	37	31	32	32
Unweighted bases		173	746	808	1,727

Base: all final survey respondents who accessed services other than the IPS intervention and who are not currently in employment.

Table 6.7: How much the support received from these services increased or decreased motivation to find employment, among trial groups

			Trial group Control %	Trial group Treatment %	Total %	P-value
Impact of support received on motivation to find employment	SCR IW	Increased a lot	17	15	16	0.635
Impact of support received on motivation to find employment	SCR IW	Increased a little	33	31	32
Impact of support received on motivation to find employment	SCR IW	Decreased a little	8	6	7
Impact of support received on motivation to find employment	SCR IW	Decreased a lot	8	8	8
Impact of support received on motivation to find employment	SCR IW	No effect on motivation	34	41	37
Impact of support received on motivation to find employment	SCR OOW	Increased a lot	19	26	22	0.000
Impact of support received on motivation to find employment	SCR OOW	Increased a little	27	38	32
Impact of support received on motivation to find employment	SCR OOW	Decreased a little	8	6	7
Impact of support received on motivation to find employment	SCR OOW	Decreased a lot	11	4	8
Impact of support received on motivation to find employment	SCR OOW	No effect on motivation	36	26	31
Impact of support received on motivation to find employment	WMCA	Increased a lot	20	21	20	0.036
Impact of support received on motivation to find employment	WMCA	Increased a little	27	34	30
Impact of support received on motivation to find employment	WMCA	Decreased a little	12	7	9
Impact of support received on motivation to find employment	WMCA	Decreased a lot	10	5	8
Impact of support received on motivation to find employment	WMCA	No effect on motivation	31	33	32
Impact of support received on motivation to find employment	All	Increased a lot	19	23	21	0.000
Impact of support received on motivation to find employment	All	Increased a little	28	35	31
Impact of support received on motivation to find employment	All	Decreased a little	10	6	8
Impact of support received on motivation to find employment	All	Decreased a lot	10	5	8
Impact of support received on motivation to find employment	All	No effect on motivation	33	31	32

Base: all final survey respondents who accessed services other than the IPS intervention and who are not currently in employment.

			Trial group Control %	Trial group Treatment %	Total
Unweighted base	SCR IW		81	92	173
Unweighted base	SCR OOW		371	375	746
Unweighted base	WMCA		419	389	808
Unweighted base	All		871	856	1,727

Base: all final survey respondents who accessed services other than the IPS intervention and who are not currently in employment.

Table 6.8: Most important barrier to working, by trial group

		Trial group SCR IW %	Trial group SCR OOW %	Trial group WMCA %	Total	P-value
Most important barrier to working	Difficulty finding a suitable job	12	15	16	14	0.029
Most important barrier to working	Availability or cost of transport to work	3	4	4	4	0.120
Most important barrier to working	Availability or cost of childcare	2	2	1	2
Most important barrier to working	Lack of qualifications or experience	11	11	15	12	0.001
Most important barrier to working	Lack of confidence in abilities or skills	15	9	9	10	0.000
Most important barrier to working	Physical health condition	18	18	20	19	0.428
Most important barrier to working	Mental health condition	23	24	19	22	0.005
Most important barrier to working	Caring for a child, or an elderly or disabled family member	3	3	2	3
Most important barrier to working	Being financially worse off	5	3	3	4
Most important barrier to working	Another reason	7	10	11	10
Unweighted base		984	1,383	1,298	3,665

Base: all final survey respondents.

7. References

Becker, DR, Swanson, SJ, Reese, SL, Bond, GR & McLehman, BM (2019) ‘Supported employment fidelity review manual: A companion guide to the evidence-based IPS Supported Employment Fidelity Scale’, The IPS Employment Center at The Rockville Institute, Westat.

Bond, GR, Drake, RE and Becker, DR (2020) ‘An update on individual placement and support’, World Psychiatry, 19(3), p.390.

Bond, GR, Peterson, AE, Becker, DR & Drake, RE (2012) ‘Validation of the revised individual placement and support fidelity scale (IPS-25)’, Psychiatric Services 63(8): 758-763.

Brinchmann, B, Widding‐Havneraas, T, Modini, M, Rinaldi, M, Moe, CF, McDaid, D, Park, AL, Killackey, E., Harvey, S.B. and Mykletun, A. (2020) ‘A meta‐regression of the impact of policy on the efficacy of Individual Placement and Support’, Acta Psychiatrica Scandinavica, 141(3), pp.206-220.

Burns, T and Catty, J (2008), ‘IPS in Europe: The EQOLISE trial’, Psychiatric rehabilitation journal, 31(4), p.313.

Burns, T, Yeeles, K, Langford, O, Montes, MV, Burgess, J and Anderson, C (2015) ‘A randomised controlled trial of time-limited individual placement and support: IPS-LITE trial’, The British Journal of Psychiatry, 207(4), pp.351-356.

Coole, C, Drummond, A and Watson, PJ (2013), ‘Individual work support for employed patients with low back pain: a randomized controlled pilot trial’, Clinical rehabilitation, 27(1), pp.40-50.

Dubrow S and Ganzach Y (2014) ‘Job Satisfaction over Time: A Longitudinal Study of the Differential Roles of Age and Tenure’, Academy of Management Annual Meeting Proceedings 2014(1): 13905-13905

DWP and DHSC, Work, Health and Disability Green Paper Data Pack, 2016

DWP and DHSC, Improving Lives The Future of Work, Health and Disability, 2017

Ferguson, KM, Xie, B and Glynn, S (2012, June) ‘Adapting the individual placement and support model with homeless young adults’, Child & Youth Care Forum, 41(3) pp. 277-294). Springer US.

Frederick, DE and VanderWeele, TJ (2019) ‘Supported employment: meta-analysis and review of randomized controlled trials of individual placement and support’, PloS-one, 14(2), p.e0212208.

Fyhn, T, Ludvigsen, K, Reme, SE and Schaafsma, F (2020) ‘A structured mixed method process evaluation of a randomized controlled trial of Individual Placement and Support (IPS)’, Implementation science communications, 1(1), pp.1-11.

Heslin, M, Howard, L, Leese, M, McCrone, PAUL, Rice, C, Jarrett, M, Spokes, T, Huxley, P and Thornicroft, G (2011) ‘Randomized controlled trial of supported employment in England: 2 year follow-up of the Supported Work and Needs (SWAN) study’, World Psychiatry, 10(2), p.132.

Howard, LM, Heslin, M, Leese, M, McCrone, P., Rice, C, Jarrett, M, Spokes, T., Huxley, P and Thornicroft, G (2010) ‘Supported employment: randomised controlled trial’, The British Journal of Psychiatry, 196(5), pp.404-411.

IPS Employment Centre (2021) ‘Individual Placement and Support (IPS): The Evidence-Based Practice for Employment’, Updated 7.27.21. As of 27/03/2022.

IPS Employment Centre (undated), [What is IPS] (https://ipsworks.org/index.php/what-is-ips/)

Kim, SJ, Bond, G.R., Becker, DR, Swanson, SJ and Langfitt-Reese, S (2015) ‘Predictive validity of the individual placement and support fidelity scale (IPS-25): a replication study’. Journal of Vocational Rehabilitation, 43(3), pp.209-216.

Latimer, EA, Lecomte, T, Becker, DR, Drake, RE, Duclos, I, Piat, M, Lahaie, N, St-Pierre, MS, Therrien, C and Xie, H (2006) ‘Generalisability of the individual placement and support model of supported employment: results of a Canadian randomised controlled trial’, The British Journal of Psychiatry, 189(1), pp.65-73.

Li-Tsang, CWP, Li, EJQ, Lam, CS, Hui, KYL. and Chan, CCH. (2008) ‘The effect of a job placement and support program for workers with musculoskeletal injuries: a randomized control trial (RCT) study’, Journal of Occupational Rehabilitation, 18(3), pp.299-306.

Lones, CE, Bond, GR, McGovern, MP, Carr, K, Leckron-Myers, T, Hartnett, T and Becker, D.R. (2017) ‘Individual placement and support (IPS) for methadone maintenance therapy patients: a pilot randomized controlled trial’, Administration and Policy in Mental Health and Mental Health Services Research, 44(3), pp.359-364.

Magura, S, Blankertz, L, Madison, EM, Friedman, E and Gomez, A (2007) ‘An innovative job placement model for unemployed methadone patients: a randomized clinical trial’, Substance use & misuse, 42(5), pp.811-828.

Marmot M, Allen J, Boyce T, Goldblatt P, Morrison J (2020) Health equity in England: The Marmot Review 10 years on. London: Institute of Health Equity;

Metcalfe, JD, Drake, RE and Bond, GR (2018) ‘Economic, labor, and regulatory moderators of the effect of individual placement and support among people with severe mental illness: a systematic review and meta-analysis’, Schizophrenia Bulletin, 44(1), pp.22-31.

Modini, M, Tan, L, Brinchmann, B, Wang, MJ, Killackey, E, Glozier, N, Mykletun, A and Harvey, SB, (2016) ‘Supported employment for people with severe mental illness: systematic review and meta-analysis of the international evidence’, The British Journal of Psychiatry, 209(1), pp.14-22.

Moran, M (2017) The 2015 ESA Trials: A Synthesis, Department for Work and Pensions, August 2017.

Newton B, Sainsbury R (2017) The 2015 ESA trials: a synthesis of qualitative research with claimants, Research Report 933, Department for Work and Pensions, Aug 2017.

Office for National Statistics (ONS), (2022) Official Statistics: The employment of disabled people 2021, Department for Work and Pensions.

Office for National Statistics (ONS), (17 May 2022) A08: Labour market status of disabled people, Department for Work and Pensions.

Oshima, I, Sono, T, Bond, G.R., Nishio, M. and Ito, J, (2014) ‘A randomized controlled trial of individual placement and support in Japan’, Psychiatric rehabilitation journal, 37(2), p.137.

Ottomanelli, L, Barnett, SD and Toscano, R (2014) ‘Individual placement and support (IPS) in physical rehabilitation and medicine: The VA spinal cord injury experience’, Psychiatric rehabilitation journal, 37(2), p.110.

Picken, N, Hofman, J, Gilder, L, Ryen Gloinson, E and Ling, T (2021) Evaluation of IPS Grow: Final report. Santa Monica, CA: RAND Corporation, 2021.

Reme, SE, Grasdal, AL, Løvvik, C, Lie, SA and Øverland, S (2015) ‘Work-focused cognitive–behavioural therapy and individual job support to increase work participation in common mental disorders: a randomised controlled multicentre trial’, Occupational and environmental medicine, 72(10), pp.745-752.

Swanson, SJ, Becker, DR, Drake, RE and Merrens, MR (2008) Supported employment: A practical guide for practitioners and supervisors. Kearney, NE: Morris Publishing.

Waddell G, Burton AK (2006) Is Work Good For Your Health And Well-Being? Department for Work and Pensions

Wallstroem, IG, Pedersen, P, Christensen, TN, Hellström, L, Bojesen, AB, Stenager, E, White, S, Mueser, K., Bejerholm, U, van Busschbach, JT and Michon, H (2021) ‘A Systematic Review of Individual Placement and Support, Employment, and Personal and Clinical Recovery’, Psychiatric Services, pp.appi-ps.

Wilson, T (2022) Labour Market Statistics, June 2022, IES briefing, Brighton, June 2022.

1. Improving Access to Psychological Therapies (IAPT) programme ↩

2. The full fidelity scale measures are shown in Chapter 4 of ‘Evaluation of the Health-led Employment Trials: Appendices to the 4-month outcomes report Appendices’. ↩

3. This means the extent to which an outcome varies. An impact of 1 standard deviation would move the average person from the 50th percentile of the distribution for that outcome to the 84th percentile. An impact of 0.1 or 0.2 standard deviations would move the median individual from the 50th percentile to the 54th or 58th percentile, respectively. ↩ ↩²

4. There was a positive trend that was not significant for SCR IW and WMCA, although a positive effect emerged on earnings in WMCA shortly after the 12-month outcomes point. ↩

5. A matched funding partnership between National Lottery Community Fund and the European Social Fund for projects across England that tackle poverty and promote social inclusion. https://www.tnlcommunityfund.org.uk/funding/programmes/building-better-opportunities ↩

6. https://www.nihr.ac.uk/documents/confirming-local-capacity-and-capability/11619 ↩

7. Formerly part of the Cabinet Office and now an independent organisation ↩

8. Please note that this differs from the number of people randomised into the trial (9,785) as some withdrew before the surveys were issued. ↩

9. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/685903/The_Green_Book.pdf ↩