Standard

Department for Health and Social Care and NHS Digital: QCovid algorithm

Published 1 June 2022

Name	Tier and category	Description	Entry (please enter all the required information in this column)
Name	Tier 1 - Overview	Colloquial name used to identify the algorithmic tool.	• COVID-19 Clinical Tool/ QCovid algorithms
Description	Tier 1 - Overview	Give a basic overview of the purpose of the algorithmic tool. Explain how you’re using the algorithmic tool, including: • how your tool works • how your tool is incorporated into your decision making process Explain why you’re using the algorithmic tool, including: • what problem you’re aiming to solve using the tool, and how it’s solving the problem • your justification or rationale for using the tool • how people can find out more about the tool or ask a question - including offline options and a contact email address of the responsible organisation, team or contact person	• The aim of the clinical tool is to create a way of predicting how at-risk individuals might be from coronavirus. The tool was developed for use by clinicians in support of conversations with patients about personal risk. • The tool utilises algorithms developed by the University of Oxford, called QCOVID. These combine a number of factors such as age, sex, ethnicity, height and weight (to calculate BMI), and specific health conditions and treatments in order to estimate the combined risk of catching coronavirus and being hospitalised or catching coronavirus and dying. • More information is available at COVID-19 Clinical Risk Assessment Tool. • The original version of the QCOVID algorithms were also used as part of the Population Risk Assessment to add patients to the Shielded Patient List in February 2021. These patients were advised to shield at that time were provided support for doing so, and were prioritised for COVID-19 vaccination. The Shielded Patient List has since been closed.
URL of the website	Tier 1 - Overview	If available, provide the URL reference to a page with further information about the algorithmic tool and its use. This facilitates users searching more in-depth information about the practical use or technical details. This could, for instance, be a local government page, a link to a GitHub repository or a departmental landing page with additional information.	• See COVID-19 Clinical Risk Assessment Toollink • See Shielded Patient List
Contact email	Tier 1 - Overview	Provide the email address of the organisation, team or contact person for this entry.	[email protected]
1.1 Organisation/ department	Tier 2 - Owner and responsibility	Provide the full name of the organisation, department or public sector body that carries responsibility for use of the algorithmic tool. For example, ‘Department for Transport’.	• Department for Health and Social Care • NHS Digital
1.2 Team	Tier 2 - Owner and responsibility	Provide the full name of the team that carries responsibility for use of the algorithmic tool.	• Commissioner: Department for Health and Social Care – Shielding / Enhanced Protection Programme • Provider: NHS Digital
1.3 Senior responsible owner	Tier 2 - Owner and responsibility	Provide the role title of the senior responsible owner for the algorithmic tool.	• Senior Responsible Owner during delivery: Phil Harper (DHSC)
1.4 Supplier or developer of the algorithmic tool	Tier 2 - Owner and responsibility	Provide the name of any external organisation or person that has been contracted to develop the whole or parts of or the algorithmic tool.	• BJSS Limited/Sparck • University of Oxford • Hippo • Kainos
1.5 External supplier identifier	Tier 2 - Owner and responsibility	If available, provide the Companies House number of the external organisation that has been contracted to develop the whole or parts of or the algorithmic tool. You can get a company’s Companies House number by finding company information or using the Companies House API	• BJSS Limited - company number 02777575 • Hippo - 09877239 • Kainos – company number 09047312
1.6 External supplier role	Tier 2 - Owner and responsibility	Give a short description of the role the external supplier assumed with regards to the development of the algorithmic tool.	• The University of Oxford developed the algorithm used for the Population Risk Assessment, and those housed within the clinical tool for clinicians. • BJSS Limited developed the architecture, processes and tools for delivering the clinical tool. • Kainos provided resources to lead on requirements and design development for the tool as well as owning the approvals process. • Hippo provided User Researchers and produced reports and guidance that informed development.
1.7 Terms of access to data for external supplier	Tier 2 - Owner and responsibility	Details the terms of access to (government) data applied to the external supplier.	• Client/supplier agreements to cover use of data. • The Statement of Work covers suppliers to NHS Digital who must be Baseline Personnel Security Standard (BPSS) confirmed. • The Oxford University team were given temporary access to central government data in order to develop the algorithm (for example, NIMS and HES) to combine with GP data already held. • Government data was not used in the development of the wider tool that housed the algorithm from Oxford University. • 12 team members (for example Data Architects and Testers) also have Disclosure and Barring Service clearance. • Read redacted version of the procurement contract.
2.1 Scope	Tier 2 - Description	Describe the purpose of the tool in terms of what it’s been designed for and what it’s not been designed for. This can include a list of potential purposes that the tool was not designed to fulfil but which could constitute possible common misconceptions in the future.	• The original version of the QCOVID algorithms were used as part of the Population Risk Assessment to add patients to the Shielded Patient List in February 2021. These patients were advised to shield at that time, were provided support for doing so, and were prioritised for COVID-19 vaccination. • The tool has been designed to allow clinicians to input information about individuals in order to calculate risk of suffering adverse outcomes in the event that an individual contracts Covid. • The tool estimates the likelihood of a particular outcome. It does not guarantee outcomes. • The current version of the tool makes use of separate algorithms for vaccinated and unvaccinated patients, but the original research methodology was not designed to establish the effectiveness of vaccines. • The absolute risk estimates produced by the tool were ultimately derived from the study period used in the research. This predates new variants of the virus and other developments in population immunity, so the main purpose of the tool remains as a guide to personal relative risk.
2.2 Benefit	Tier 2 - Description	Describe the key benefits that the algorithmic tool is expected to deliver, and an expanded justification on why the tool is being used.	• The key benefits of the tool when it was used in the Population Risk Assessment was to identify individuals at a high risk who should  be advised to shield and/or be placed into a high priority cohort for the vaccine rollout. The ultimate aim was to ensure protective interventions were targeted at the most vulnerable, to reduce mortality and protect the NHS. • The clinical tool for clinicians is intended to support individual conversations with patients about risk. Originally, the goal was to help patients understand the reasons for being asked to shield and, where relevant, help them do so. Since the end of shielding requirements, it is hoped that better-informed conversations about risk will have supported patients to make appropriate decisions about personal risk, either protecting them from adverse health outcomes or to some extent alleviating concerns about re-engaging with society. • Furthermore, the technology introduced a new capability, created at speed during the early stages of the pandemic, as it adds input to aid more targeted clinical interventions. • The underlying principles that drive the tool may in the future lend themselves to application outside of COVID-19, particularly given that the technology has been proven in a real world application.
2.3 Alternatives considered	Tier 2 - Description	Provide, where applicable, a list of non-algorithmic alternatives considered, or a description of how the decision process was conducted previously.	• Before the development of QCOVID, the Shielded Patient List was developed by identifying patients for protection on the basis of whether or not they had any of a list of medical conditions. This did not take into account demographic factors or combinations of co-morbidities which QCOVID was then able to accommodate in its decision-making process. • England’s Chief Medical Officer commissioned a predictive model for a data-driven approach to COVID-19 clinical risk assessment which resulted in the development of QCOVID.
2.4 Type of model	Tier 2 - Description	Indicate which types of methods or models the algorithm is using. For example, expert system, deep neural network and so on.	• In essence, the tool creates a risk calculation based on scoring risk factors across a number of data fields pertaining to demographic, clinical and social patient information. This process is detailed below: • The algorithm creates a risk prediction model - QCovid - that provides a weighted, cumulative calculation of absolute risk using the variables associated with poor COVID-19 outcomes.  The research on which the model is based uses a Cox proportional hazards survival model using a range of health and demographic variables, excluding those not significant from final models used in decisions. More information on the research behind the current versions of QCOVID. • The factors incorporated in the model include age, ethnicity, level of deprivation, obesity, whether someone lived in residential care or was homeless, and a range of existing medical conditions, such as cardiovascular disease, diabetes, respiratory disease and cancer. • For the latest clinical tool, separate versions of the QCOVID models were estimated for vaccinated and unvaccinated patients. When using the clinical tool, clinicians enter patients’ vaccination status, and the tool uses this to determine which algorithm will be used in the calculations.
2.5 Frequency of usage	Tier 2 - Description	Provide information on how regularly the algorithmic tool is being used. For example the number of decisions made per month, the number of citizens interacting with the tool, and so on.	• The Clinical Tool has been used within and outside of the NHS. In the period between 1st January 2022 and 31st March 2022, there were 2,180 completed assessments. Increased uptake was driven, in part, due to increased access, as technical restrictions on who could access the tool were removed in January 2022. • Assessment numbers often move with relative infection rate (e.g. higher infection rate leads to more usage of the tool).
2.6 Phase	Tier 2 - Description	Describe the phase in which of the following stages or phases the tool is currently situated: - idea - design - development - production - retired This field includes date and time stamps of creation and any updates.	Production/run and maintain
2.7 Maintenance	Tier 2 - Description	Give details on the maintenance schedule and frequency of any reviews. For example, specific details on when and how a person reviews or checks the automated decision.	• Maintenance schedules for the algorithm are dictated by the emerging understanding of COVID and any new variants and therefore do not fit into a routine schedule. • Changes are processed through a RFC (request for change) procedure and enacted by the NHS Digital delivery team. • To date, maintenance has included the addition of extra data points to the algorithm (e.g. Vaccine status).
2.8 System architecture	Tier 2 - Description	If available, provide the URL reference to documentation about the system architecture. For example, a link to a GitHub repository image or additional documentation about the system architecture.	See graph.
3.1 Process integration	Tier 2 - Oversight	Explain how the algorithmic tool is integrated into the decision-making process and what influence the algorithmic tool has on the decision-making process. Give a more detailed and extensive description of the wider decision-making process into which the algorithmic tool is embedded.	• The use of the tool does not override any clinical decision making but is a supporting device in the decision making process. • The tool amalgamates known factors about Covid at the time the tool was released. The update schedule for the tool isn’t confirmed as it is dependent on scientific understanding of new variants as they emerge. • The tool promotes shared decision making with the patient and is an extra point of information to consider in the decision making process. • The tool helps with risk/benefit analysis around decisions (e.g. recommendation to shield or take other precautionary measures). • The tool is treated, in essence, as a clinical device (class 1) to aid decision making, not to make automated decisions by itself. • As of April 2022, there are no further planned updates to the tool.
3.2 Provided information	Tier 2 - Oversight	Describe how much and what information the algorithmic tool provides to the decision maker.	Example outputs from the tool are below: Risk of dying of COVID-19 following a positive PCR test Absolute risk: 0.0102% We’d expect that 10 out of 100,000 people with the same characteristics and risk factors as this patient would die of COVID-19 following a positive PCR test. Relative risk: 1.1 We’d expect that the patient is 1.1 times more likely to die of COVID-19 following a positive PCR test than someone who shares the same age and sex registered at birth as them, but none of their other characteristics and risk factors. Result generated on 25 February 2022 and valid for 90 days. Release R1 Version 2.0.5.9292
3.3 Human decisions	Tier 2 - Oversight	Describe the decisions that people take in the overall process, including human review options.	• The tool is designed to help guide discussions but does not mandate the treatment or next steps agreed for the patient. • The tool only provides one strand of information as guidance. In order to make a decision on vaccination or treatment, many strands of information to determine those who are eligible would be required.
3.4 Required training	Tier 2 - Oversight	Describe the required training those deploying or using the algorithmic tool must undertake, if applicable; For example, the person responsible for the management of the tool had to complete data science training.	• The tool is designed for use by clinicians who are reminded to look through clinical guidance before using the tool
3.5 Appeals and review	Tier 2 - Oversight	Provide details on the mechanisms that are in place for review or appeal of the decision available to the general public.	• Patients will first review results with their clinician and for any further clarification can refer to Guidance for clinicians about the COVID-19 Clinical Risk Assessment Tool.
4.1 Source data name	Tier 2 - Information on data	If applicable, provide the name of the datasets used.	• Data is entered by the user and is not stored in a database. No external datasets are used/loaded. • Townsend deprivation index is looked up by postcode and fed into algorithm (provided by the team at Oxford University) • Current daily infection rate for England is taken from the UK Coronavirus dashboard.
4.2 Source data	Tier 2 - Information on data	Give an overview of the data used to train and run the algorithmic tool. It will also specify whether data is used for training, testing, or operating. It should include which categories of data - for example ‘age’ or ‘address’ - which were used to train the model and which are used as input data for making a prediction.	The team used the QResearch database (version 46) of 12 million patients with personal, clinical, and drug data that have been used for clinical and drug safety research. QResearch is linked to multiple datasets at the individual patient level. For this analysis, we used the National Immunisation Database of Covid-19 vaccinations to identify data on vaccine date and doses for all people vaccinated in England. For hospital admissions, we used the linked Hospital Episode Statistics dataset supplemented by the more regularly updated Secondary Users Service data. We also obtained and linked the following datasets: national data for mortality; SARS-CoV-2 infection; systemic anticancer treatment; radiotherapy treatment datasets; and national cancer registry data.
4.3 Source data URL	Tier 2 - Information on data	If available, provide a URL to the dataset.	n/a
4.4 Data collection	Tier 2 - Information on data	Gives information on the data collection process, including the original purpose of data collection.	• No data is collected other than by the clinician at point of use • The algorithm undertakes a calculation on patient data and the result of that calculation is retained by postcode. Patient level data is not retained. • Data is not patient identifiable at the point of input.
4.5 Data sharing agreements	Tier 2 - Information on data	Provide further information on data sharing agreements in place.	• Anonymised data is stored in the logs for reporting purposes (i.e. patient data input by clinician – age, gender, risk factors, etc.).
4.6 Data access and storage	Tier 2 - Information on data	Provide details on who has or will have access to this data, how long it’s stored, under what circumstances and by whom.	Management Information (MI) is retained for 90 days and then a summary version is retained for 8 years. This is only available to approved NHS Digital staff and access is controlled in line with the following governance standards: • SO 13485:2016: Medical devices - Quality management systems - Requirements for regulatory purposes. • DBC0129: Clinical Risk Management: its Application in the Manufacture of Health IT Systems. • DCB0160: Clinical Risk Management: its Application in the Deployment and Use of Health IT Systems • IEC 62304:2006 Amd 1:2015: Medical device software – Software lifecycle processes.
5.1 Impact assessment name	Tier 2 - Risk mitigation and impact assessment	Provide the name and a short overview of the impact assessment conducted.	• A clinical risk assessment in accordance with mandated Safety Standards DCB0129: Clinical Risk Management and MDD Medical Device Directive 93/32EEC.
5.2 Impact assessment description	Tier 2 - Risk mitigation and impact assessment	Give a description of the impact assessments conducted.	• Risk Assessment conducted considered Risk Analysis, Risk Evaluation, Risk Control and implementation. • In accordance with DCB0129: a Safety Case Report and Hazard Log describing the hazard(s) scenario, understanding risk impact and suitable mitigation to alleviate risks through design, test, business process.
5.3 Impact assessment date	Tier 2 - Risk mitigation and impact assessment	Provide the date in which the impact assessment was conducted.	• A risk assessment is an iterative process throughout the lifecycle of the tool. As new software components are introduced, fresh assessments are undertaken, and the Hazard Log is iteratively updated to align with the latest version of the software.
5.4 Impact assessment link	Tier 2 - Risk mitigation and impact assessment	If available, provide a link to the impact assessment.	Example outputs from the Hazard log (when the shielding programme was active) are included below: • Hazard: A person who should not be identified as meeting the threshold for higher risk using QCOVID are identified inaccurately • Cause: Information held in GP/hospital records is clinically inaccurate, out-of-date or not recorded, i.e. poor quality of coding. • Effect: Output from COVID-19 Predictive Risk Model misidentifies person as meeting the threshold for higher risk
5.5 Risk name	Tier 2 - Risk mitigation and impact assessment	Provide an overview of the common risks for the algorithmic tool.	• As part of patient safety risk assessment, Hazardous scenarios are documented, yet haven’t occurred as suitable mitigation is introduced and implemented to alleviate the risk.
5.6 Risk description	Tier 2 - Risk mitigation and impact assessment	Give a description of the risks identified.	Example potential scenarios include: • Hazard - COVID-19 Predictive Risk Model Tool not generated for a Patient when expected. • Hazard - Patient not provided with the most accurate risk score (either excessive or insufficient score) from the algorithm. • Patient does not have an output generated from the COVID-19 Predictive Risk Model
5.7 Risk mitigation	Tier 2 - Risk mitigation and impact assessment	Provide an overview of how the risks have been mitigated.	• Defined within the Hazard Log. In summary, each hazard identified is reviewed, risk classified and- through design, test, business process or training- the appropriate mitigations are defined and implemented. • Hazard log inputs outlined in 5.4