Guidance

PIP statistics: quality assurance of administrative data report

Published 13 June 2023

The latest release of Personal Independence Payment statistics can be found in the Personal Independence Payment statistics collection.

1. Introduction

1.1 Background

This report contains information on the Personal Independence Payment (PIP) administrative data sources used by the Department for Work and Pensions (DWP), as well as quality assessments on each of them.

The UK Statistics Authority have published a regulatory standard including a Quality Assurance of Administrative Data (QAAD) toolkit. The standard was developed in response to concerns about the quality of administrative data and in recognition of the increasing role that such data is playing in the production of official statistics.

1.2 List of Administrative Data Sources

Data from 2 administrative systems are used in the production of PIP official statistics:

• Personal Independence Payment Computer System (PIP CS)
• Customer Information System (CIS)

A brief summary of these systems and their uses is described below.

Personal Independence Payment Computer System (PIP CS)

Personal Independence Payment Computer System is the main system for the administration of PIP claims. It is used as the primary data source for PIP official statistics to determine who is claiming PIP and records the details of their claims throughout the PIP process.

Customer Information System (CIS)

The Customer Information System contains a record for all individuals who have registered for PIP and been issued with a National Insurance number. The CIS is used to capture personal information (such as age, sex and geographic location) about benefit claimants. It is used as a data source for claimants’ geographic data and date of death for the purposes of producing PIP statistics.

2. Quality assurance of administrative data (QAAD) assessment

2.1 UK Statistics Authority QAAD toolkit

The assessment of the PIP administrative data sources has been carried out in accordance with the QAAD toolkit.

The QAAD toolkit sets out four levels of quality assurance that may be required of a dataset:

• A0 – no assurance
• A1 – basic assurance
• A2 – enhanced assurance
• A3 – comprehensive assurance

The UK Statistics Authority states that the A0 level is not compliant with the Code of Practice for Statistics. The assessment of the assurance level is in turn based on a combination of assessments of data quality risk and public interest. The toolkit sets out the level of assurances required as follows:

Level A1 – basic assurance

The statistical producer has reviewed and published a summary of the administrative data quality assurance (QA) arrangements.

Level A2 – enhanced assurance

The statistical producer has evaluated the administrative data QA arrangements and published a fuller description of the assurance.

Level A3 – comprehensive assurance

The statistical producer has investigated the administrative data QA arrangements, identified the results of independent audit and published detailed documentation about the assurance and audit.

To determine which assurance level is appropriate for a statistics publication it is necessary to take a view of the level of risk of quality concerns and the public interest profile of the statistics.

Each administrative data source has been evaluated according to the toolkits risk and profile matrix (Table 1), reflecting the level of risk to data quality and the public interest profile of the statistics.

Table 1: UK Statistics Authority quality assurance of administrative data (QAAD) risk and profile matrix

	Lower public interest profile	Medium public interest profile	Higher public interest profile
Low level of risk of quality concerns	Statistics of lower quality concern and lower public interest [A1]	Statistics of low quality concern and medium public interest [A1 or A2]	Statistics of a low quality concern and higher public interest [A1 or A2]
Medium level of risk of quality concerns	Statistics of medium quality concern and lower public interest [A1 or A2]	Statistics of medium quality concern and medium public interest [A2]	Statistics of medium quality concern and higher public interest [A2 or A3]
High level of risk of quality concerns	Statistics of higher quality concern and lower public interest [A1 or A2 or A3]	Statistics of higher quality concern and medium public interest [A3]	Statistics of higher quality concern and higher public interest [A3]

Source: Office for Statistics Regulation

2.2 Assessment and justification against the QAAD risk and profile matrix

The data risk of quality concern and public interest profile in PIP statistics are rated by assessing:

• (a) the possibility of quality concerns arising in the administrative data that may affect the statistics’ quality
• (b) the nature of the public interest served by the statistics

(a) The PIP data are regarded as being a medium risk of data quality concern. While every effort is made to collect data to the highest quality, as with all administrative data it is dependent on the accuracy of information entered into the system. Checks are made throughout the process from collection of the data to producing the statistics but some data entry or processing errors may filter through.

(b) The PIP official statistics are regarded as higher public interest due to regular coverage of PIP policies and statistics in the media and their impact on the lives of vulnerable UK citizens.

Therefore, as defined by the risk and profile matrix (Table 1), the combination of medium level of data risk concerns, and higher public interest profile indicate that enhanced assurance [A2] is the minimum level required for PIP statistics.

The QAAD toolkit outlines 4 specific areas for assurance, and the rest of this report will focus on these areas in turn. These are:

• operational context and administrative data collection
• communication with data supply partners
• quality assurance principles, standards and checks applied by data suppliers
• producer’s quality assurance investigations and documentation

Each of the 4 practice areas are evaluated separately, and the respective level of assurance is stated. This approach enabled an in-depth investigation of the areas of particular risk or interest to users.

The overall level of assurance for PIP statistics is outlined in the summary section.

3. Areas of quality assurance of administrative data (QAAD)

3.1 Operational context and administrative data collection (QAAD matrix score A2)

This relates to the need for statistical producers to gain an understanding of the environment and processes in which the administrative data are being compiled and the factors that might increase the risks to the quality of the administrative data.

Personal Independence Payment Computer Systems (PIP CS) Data Collection

The information needed to produce PIP statistics is mainly obtained from the administrative systems that are used to operate PIP. The accuracy and completeness of data at this stage is crucial to the quality of the statistics.

The data on the PIP CS administrative system is built up from information entered by DWP decision-makers, Assessment Providers and other members of staff working on PIP, inputting the information they receive from those seeking to claim PIP. These administrative data are then processed through automated procedures into a form that is usable for the production of PIP statistics.

The first stage of claiming PIP is largely a telephone-based service with most claimants starting their claim through calling the ‘PIP new claims’ phone line. It is possible for claimants to apply for PIP by post, although this takes longer. This stage is followed by a PIP 2 questionnaire (this can be either paper-based or online) and then in some cases a face-to-face assessment with an Assessment Provider.

The following processes detail the information that is collected during a claimant’s journey through PIP and the verification procedures in place.

Submitting a claim

To make an application, claimants call the ‘PIP new claims’ phone line. They are asked questions to collect information about their circumstances and submit their claim. Alternatively, they can make their application by post.

If a claim fulfils the eligibility criteria and passes any management checks, a ‘PIP2’ questionnaire is then completed to collect more information regarding the nature and impact of the claimant’s disability or disabilities. An Assessment Provider (AP) will then evaluate their claim. This may involve the claimant being invited to an interview so that the AP can assess their ability to carry out tasks. This interview can take place via phone, video or in person. The AP will make a recommendation, and then the DWP decision-maker will make a decision about:

• whether to award the claimant PIP or not
• the type of award to give the claimant (daily living component, mobility component or both)
• the level of the component(s) to give the claimant (enhanced or standard)
• the duration of the award and length of review period if relevant

At each of these stages, the claimant’s data are inputted into the PIP CS, so that their information is updated as their claim is processed.

Once a claim has ended (due to expiration of the PIP award, failure to pass an award review, permanent absence from the country, or the death of the claimant), an off-flow event is recorded in the PIP CS.

Strengths

• Information is updated in a timely manner
• Information is verified either independently or by documentary evidence provided by the claimant
• Coverage is expected to be close to 100% given the nature of PIP CS data collection
• Data quality is improved by using data from other systems (the CIS)

Weaknesses

• Developments in the PIP CS system present risks to the consistency of data collection
• Technical issues affecting PIP CS system and underlying hardware/software can also present risks to data collection
• As with any administrative system, there is potential for fraud and administrative error, as the process relies on information submitted by claimants and verified by operational teams
• Data is subject to revisions that may take several months to filter through

3.2 Communication with data supply partners (QAAD matrix score A2)

This relates to the need to maintain effective relationships with suppliers (through written agreements such as service level agreements or memoranda of understanding), which include change management processes and the consideration of statistical needs when changes are being made to relevant administrative systems.

PIP data are owned by DWP and provided to analysts as a business requirement for management information (MI), policy analysis and statistics.

The PIP CS was built externally but managed by an internal team within DWP. This team are the principal data supply partners for PIP official statistics. They are responsible for providing PIP data to the analytical community within DWP to agreed specifications.

A separate internal data handling team are responsible for processing the data into a format that is useable by analysts. They liaise with the data supply team on behalf of analysts regarding data issues and proposed system changes.

The data handling team automatically receives a feed of data (100 data files three times per day) from the PIP CS. They then transform it into a more coherent configuration within the data warehouse (the servers where data are stored) called the PIP Atomic Data Store (PIP ADS), from which MI is taken. For example, they identify concepts that need to be analysed (such as a registration) and create data items relating to specific dates and events based on raw admin data records.

The data handling team ensure the data are secure by abiding by security and GDPR protocols and restricting access to the data. They also encrypt the PIP data to hide the identities of the individuals.

After processing the data and making sure it is secure, the data handling team then make the PIP ADS available to statistics producers.

The data supply team and the data handling team:

• collect, store and transfer data lawfully
• ensure the data is accurate and updated in a timely manner
• comply with internal policy to mask personal identifiers to ensure that individuals cannot be identified

There is a Data Transfer Protocol in place to clarify and guide the process of transforming the raw data into a useable asset.

Communication with the PIP data supply team

The data handling team have established communication channels with the data supply team (for resolving incidents).

If there is an issue with the data:

• The data handling team liaise with the data supply team to investigate
• Any changes that are made are carefully tested before the new data are released into the live environment

Communication with the PIP data handling team

The data handling team continuously inform the data supply team of issues with the PIP ADS and provide relevant feedback where required.

A report is made to the DWP security advice centre if the data handling team later identify an issue with the data.

The data handling team sign-off arrangements for the PIP SAS datasets to:

• ensure SAS datasets match the specifications on the quality control sheet
• report issues to the data supply team for further investigation
• ensure any identified issues are resolved

Communication with operational teams

Communication with operational and other analytical teams ensure an understanding of PIP system changes that impact on the official statistics. This is particularly important for PIP because, as a relatively new benefit, the administrative systems are constantly developing.

Once the data are extracted out of the PIP ADS, the statistical producers use them to create quarterly PIP statistical releases. The statistical producers have regular contact with a range of stakeholders, including operations. The statistical producers use the QA group and stakeholder group meetings as the main mechanisms for communicating with operations.

Strengths

• Stakeholders receive regular communication from those who communicate with data supply partners (the data delivery team and the operational teams)
• The data supply team receive feedback during the data transformation process from experts in the data (such as analysts, the data handling team and operational teams), which therefore increases the likelihood of issues being identified and resolved
• Awareness of PIP CS system changes contribute to increasing the accuracy of the data

Weaknesses

• The statistics team are not involved in PIP CS system design and have no direct engagement with the system designers
• There is a small production window to create the PIP SAS datasets and issues are likely to cause delays

3.3 Quality assurance principles, standards and checks by data supplier (QAAD matrix score A2)

This relates to the validation checks and procedures undertaken by the data supplier, any audit of the operational system and any steps taken to determine the accuracy of the administrative data.

The data supply team carry out quality assurance checks throughout the data transformation process.

Data Supplier QA checks

There are systems in place to quality assure data as it is inputted into the PIP CS by DWP decision-makers.

When a person makes a PIP claim, a DWP decision-maker will ask security questions to verify the claimant’s identity. If these security questions produce invalid responses, the case closes immediately. Otherwise, the DWP decision-maker will continue to gather information from the claimant, and fill in the claimant’s responses into the PIP CS. Various automatic validation checks are in place while this is happening, to prevent, for example, the DWP decision-maker accidently inputting a nonsensical date, or a field being left blank. If there are any claims that do not fulfil the eligibility criteria (for example due to nationality or age), the claim will no longer be processed.

If the claimant has made a claim from DWP previously, their personal details from CIS will be automatically populated into the relevant fields, although this can be updated if necessary. This further reduces the scope for error.

Once the claim has been verified to ensure it has met the eligibility criteria, around 20% of claims are randomly highlighted for management checks. Additionally, a claim might be highlighted for a management check if there is something unusual about their claim, for example, the claimant has lived abroad at all during the last three years. Claims that undergo management checks are reviewed manually, and, if required, more information is collected from the claimant.

While these checks are primarily aimed at identifying potential fraud, they also serve to check any data entry errors made by DWP decision-makers.

Data handling team QA checks

The data handling team used a series of checks at every stage when developing the PIP ADS to ensure that it would remain an accurate and robust means of recording PIP activity. In addition, any update or change is carefully checked before the data are released into the live environment. Any inconsistencies or issues raised are investigated to identify what is causing them.

Strengths

• Consistent checks allow understanding of observed differences
• Many parts of the process are automated (including data entry, initial checks, and data processing) which reduces human error
• In-depth understanding of data ensure issues are identified and resolved efficiently

Weaknesses

• While some checks are automated, there is still scope for errors when entering the data, which, if not identified, can remain in the data and lead to incorrect statistics

3.4 Producers quality assurance investigations and documentation (QAAD matrix score A2)

This relates to the quality assurance conducted by the statistical producer, including corroboration against other data sources.

The PIP ADS is refreshed automatically on a daily basis. Whilst there were numerous checks made to the data when the data handling team created the ADS, no checks are made on a day-to-day basis. However, when the statistics team produce the quarterly PIP statistics, they do a series of checks to ensure the data look realistic.

Pre-processing checks

The statistics team generates observation counts for each dataset to ensure missing data are identified at the earliest opportunity and rectified before data processing starts. The back series counts for each PIP SAS dataset are expected to match those recorded in previous months, or relatively small changes due to revisions. If there are significant changes, this may indicate missing or erroneous data and is reported to statistical producers’ QA group, who then will raise it with the data handling team for further investigation.

The variables in each dataset are compared to previous months to identify changes and therefore reduce the risk of errors during data processing. If there is a change, it is resolved by investigating, and, if required, modifying the code in relevant data processing projects.

Data processing checks

The statistics team check that each stage of data processing has completed as intended. Generally, the checks ensure that all expected variables are populated and that total observation counts are comparable to previous months. A substantial portion of data processing behind these checks are automated, reducing the likelihood of human error.

Output validation

Once the statistical outputs have been created, a series of robust quality assurance checks are carried out by the statistics team. This includes:

• checking observation counts compared to the back series for several breakdowns such as age, sex and geographic location
• checking that revisions to provisional and historic data are reasonable
• checking monthly percentage changes and proportions compared to recent trends
• investigating unexpected trends and sharp changes

In addition, the statistical outputs are compared to PIP management information (MI) data. Corroboration with MI data allows comparison of expected trends and observation counts.

The statistics team also check statistical outputs with teams across the PIP process, including:

• Operations
• Service Delivery
• Policy

This ensures that the output produced by the statistics team has been checked by multiple analytical and policy experts who are familiar with PIP data, processes and policy.

Strengths

• A pre-emptive approach ensures that issues are addressed prior to data processing. Resources within the statistics team are therefore optimised to complete other time sensitive data processing tasks
• Pre-processing checks are in place to identify missing historic data
• Extensive quality assurance throughout the statistical production process is partially automated to reduce error but includes human judgement and decision-making
• A broad range of analytical and policy experts validate statistical outputs and any changes that are made

Weaknesses

• Quality assurance is partially manual so human error is a risk factor
• The identification of a data issue during pre-processing checks may cause a delay in data processing (and possibly on timely publication)

4. Summary

DWP considers the main strengths of the PIP CS data to be that:

• information in the PIP CS is updated in a timely manner
• some information is verified by automated checks in addition to a sample of management checks
• the feedback that data supply partners receive during the data transformation process from the data handling team increases the likelihood of issues being identified and resolved
• the statistics team carry out extensive quality assurance throughout the statistical production process

The current limitations are that:

• there are potential risks to the consistency of data collection when any developments occur
• there is the potential for fraud and administrative error as the system relies on the information submitted by claimants and verification by operational teams
• the statistics team are not involved in the PIP CS design and have no direct engagement with the system designers

PIP official statistics are assessed as being assured to level A2 (enhanced assurance) as outlined by the UK Statistics Authority QAAD toolkit.

This document is accurate as of May 2023, but will be reviewed and updated periodically.

If you are of the view that this report does not adequately provide this level of assurance, or you have any other feedback, please contact us via email at [email protected] with your concerns.