Notice

Notice of Intention to impose a Monetary Penalty in respect of Pearson’s reviews of marking in 2016, 2017, 2018 and 2019

Published 31 August 2022

Applies to England

© Crown copyright 2022

This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: [email protected].

Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.

This publication is available at https://www.gov.uk/government/publications/regulatory-actions-against-pearson/notice-of-intention-to-impose-a-monetary-penalty-in-respect-of-pearsons-reviews-of-marking-in-2016-2017-2018-and-2019

Notice

1. The Office of Qualifications and Examinations Regulation (“Ofqual”) hereby gives notice that it intends to impose a Monetary Penalty on Pearson Education Ltd (“Pearson”) in the sum of £1,200,000.

2. This Notice relates to the following breaches of Pearson’s Conditions of Recognition, in respect of its reviews of marking in 2016, 2017, 2018 and 2019:

  1. (a) GCE / GCSE Conditions 17.6 (a) and (b)[footnote 1]
  2. (b) General Condition A5.2(a)
  3. (c) General Condition A5.2(e)
  4. (d) General Condition A6.1

3. Further information about Ofqual’s statutory powers and the Conditions of Recognition are set out in Annex A of this notice[footnote 2]

4. This Notice should be read in conjunction with the undertaking[footnote 3] provided by Pearson to Ofqual on 8 June 2020, in which Pearson agreed to pay compensation to affected Centres and perform the actions set out in its Action Plan.

Executive summary

5. During 2016, 2017, 2018 and 2019, Pearson failed to ensure that all of its reviews of marking in respect of its GCE and GCSE qualifications were carried out in their entirety by a person who had not been involved in the original marking[footnote 4]. In some of these cases, Pearson also failed to ensure that the reviews of marking were carried out by someone who had no personal interest in the outcome of the review[footnote 5].

6. These failings affected an estimated 46,797 reviews of marking, in part or in whole, over a 4 year period. This represents around 9% of all of the reviews carried out by Pearson each year. Around 97% of the affected reviews involved individual, anonymised answers that were reviewed at an item level on-screen. The remaining approximately 3% involved reviews of whole scripts.

7. These failings occurred as a result of the fact that:

  1. (a) Pearson did not retain a Workforce of appropriate size[footnote 6] in that it did not retain sufficient numbers of examiners to enable it to carry out its reviews of marking in a way that was compliant with the Conditions.
  2. (b) Pearson did not maintain appropriate systems of planning and internal control[footnote 7] in circumstances where -
    1. (i) Pearson failed to take steps to alter its processes to ensure compliance with the Conditions of Recognition when changes were introduced in 2016, and
    2. (ii) Pearson had been knowingly allocating reviews to examiners with previous involvement in the original marking and failed to put adequate controls in place to guard against the risks that the Conditions seek to avoid,
  3. (c) Pearson failed to take all reasonable steps to identify the risk of an incident occurring which could have had an Adverse Effect[footnote 8]. Pearson did not identify the risk of non-compliance, even when the issue was specifically drawn to its attention by Ofqual in 2018. Opportunities to identify and remedy the problem were missed, resulting in a further year of non-compliance in 2019 with 11,645 affected reviews that could have been avoided.

8. There is no evidence to show that these failings resulted in Learners or Centres receiving the wrong outcome. Nearly all affected reviews (around 99%) were conducted by Pearson’s most senior examiners who had received training. Their reviews were monitored and sampled during the review process to ensure marking standards were maintained.

9. However, these were serious breaches of Conditions that are integral to the effectiveness and purpose of the system of reviewing marking. The failures therefore have the potential to seriously undermine public confidence in the review of marking system, and the qualifications system more generally.

10. Pearson has provided an undertaking to Ofqual in which it has agreed to:

  1. (a) compensate affected Centres in the form of credit notes to the value of £320,510 to represent the income that Pearson received from affected reviews[footnote 9]; and
  2. (b) complete the actions set out in its Action Plan to ensure compliance with the Conditions for all future exam series.

11. Ofqual is assured by the undertaking and Action Plan that Pearson’s arrangements for reviews of marking for the autumn 2021 exam series and beyond have been, and will be, compliant with the Conditions of Recognition.

12. Ofqual has decided to give notice that it intends to impose a Monetary Penalty of £1,200,000 on Pearson for these breaches.

13. This decision should be viewed in the context that Ofqual accepted a Settlement Proposal from AQA in January 2020 to pay a Monetary Penalty of £350,000 in respect of similar failings with its review of marking arrangements. Ofqual’s Enforcement Committee considered that the Monetary Penalty it is intending to impose on Pearson should be significantly higher than that imposed on AQA for the following (summarised) reasons[footnote 10]:

  1. (a) This case is factually more serious than the AQA case;
  2. (b) Pearson is not entitled to any settlement discount on the level of the penalty;
  3. (c) Pearson’s undertaking to pay compensation to Centres does not include a figure to reflect its avoided compliance costs;
  4. (d) Pearson’s size and turnover is significantly greater than AQA’s.

14. Pearson and interested parties may make representations in respect of this Notice.

Background information

Types of marking

15. Pearson marks student material for GCSE and A level qualifications in 2 different ways:

  1. (a) Item level marking - this is where responses to individual questions (‘items’) are presented to markers. In this way, markers may mark thousands of student answers to the same question and in many cases, they are marking single word or other short answers.
  2. (b) Whole script marking - this is where whole exam papers (‘scripts’) are presented to markers. In this way, makers view all of a student’s answers on the exam script.

16. Centres can request a review of marking (for either method used). Pearson charges a fee for those services, unless an error is found resulting in a grade change.

Changes to reviews of marking

17. Between 10 December 2015 and 11 March 2016, Ofqual consulted on proposed changes to marking reviews, appeals, grade boundaries and the Code of Practice for GCSEs, AS and A levels. Part B of the consultation proposed to introduce new Qualification Level Conditions for reviews of marking or moderation and appeals. This included a proposal to introduce a Condition that when carrying out a review of marking of marked assessment material, an exam board (referred to as an awarding organisation in Ofqual’s Conditions) must “make sure that reviews of marking are carried out by competent people who have no personal interest in the outcome, and who have not previously been involved with the marking of the assessment.[footnote 11]

18. The table at appendix 1 of the consultation summarised the key proposals and changes, explaining that the reason for this proposed change was that “markers who review their own marking or moderation may find it difficult to be objective. They are, in any event, likely to be perceived to be biased.[footnote 12] It explained that the key change to the existing arrangements was that “the Code of Practice requires that, wherever possible, the review of marking is undertaken by someone other than the original marker (emphasis added).[footnote 13]

19. In the analysis of consultation responses, Pearson is listed as one of the non-confidential responding organisations.

20. On 26 May 2016, Ofqual published the outcome of the consultation and this change was incorporated into the new Qualification Level Conditions which were published on 15 July 2016 and came into force on 25 August 2016.

21. Any reviews of marking conducted after 25 August 2016 were required to be conducted in compliance with the new Qualification Level Conditions.

Summary of facts

Event Notification

22. On 19 November 2019, Pearson submitted an event notification to Ofqual to report that it had found that some scripts for GCSE and A level examinations that had been reviewed in its post-results service (“PRS”) window in 2016, 2017, 2018 and 2019 had been marked originally, in part or in full, by the same examiner who had then conducted the review of marking.

Pearson’s Summary of the Review of Marking issues

23. On 6 December 2019, Pearson submitted a report to Ofqual summarising the review of marking issues, providing additional information in relation to the event notification.

24. Pearson explained that on 11 November 2019, the Pearson Appeal Office had upheld an appeal from a Centre on the basis that the original examiner had been involved in the review of marking. Pearson explained that:

“…the action taken by Ofqual against AQA and OCR made it clear that the Conditions in relation to the allocation of Post Results Reviews of Marking (ROM) completely rule out the allocation of reviews to an examiner who has already marked or checked some or all of the script previously. Until now, we had believed our processes for allocating scripts were compliant with the revised conditions, in particular GCE and GCSE (9 to 1) Qualification Level Conditions and Requirements 17.6 (b).”

25. In light of this, Pearson began an investigation into the number of other reviews of marking which had been conducted, in full or in part, by the original examiner (“the affected reviews”).

Volume of affected reviews

26. A summary of the volume of affected reviews for each series is set out below[footnote 14]:

2019 series

27. Following the 2019 exam series, Pearson conducted 136,859 reviews of marking in respect of its GCSE and A level qualifications. Of those, 11,645 applications have been identified by Pearson as having been reviewed by an individual who had originally marked some, or all, of the items subject to review. The figure of 11,645 represents approximately 8.5% of all review applications made that year.

28. The applications in question can be broken down as follows:

Type of review Qualification Number of affected reviews Number of affected reviews where a fee was charged (because no grade change)
Whole script A level 142 127
Whole script GCSE 119 105
Item Level[footnote 15] A level 3834 2970
Item Level[footnote 16] GCSE 7550 5482
Total for 2019   11,645 8,684

2018 series

29. Following the 2018 exam series, Pearson conducted 128,366 reviews of marking in respect of its GCSE and A level qualifications. Of those, 11,841 applications have been identified by Pearson as having been reviewed by an individual who had originally marked some, or all, of the items subject to review. The figure of 11,841 represents approximately 9.2% of all review applications reviewed that year.

30. The applications in question can be broken down as follows:

Type of review Qualification Number of affected reviews Number of affected reviews where a fee was charged (because no grade was given)
Whole script GCSE 91 67
Item Level A level 3,431 2,641
Item Level GCSE 7,923 6,041
Total for 2018   11,841 9,081

2017 series

31. Following the 2017 exam series, Pearson conducted 129,144 reviews of marking in respect of its GCSE and A level qualifications. Of those, 11,679 applications have been identified by Pearson as having been reviewed by an individual who had originally marked some, or all, of the items subject to review. The figure of 11,679 represents approximately 9.0% of all review applications reviewed that year.

32. The applications in question can be broken down as follows:

Type of review Qualification Number of affected reviews Number of affected reviews where a fee was charged (because no grade change)
Whole script A level 158 145
Whole script GCSE 102 82
Item Level A level 3,224 2,586
Item Level GCSE 8,195 6,693
Total for 2017   11,679 9,506

2016 series

33. Following the 2016 exam series, Pearson conducted 124,782 reviews of marking in respect of its GCSE and A level qualifications. Of those, 11,632 applications have been identified by Pearson as having been reviewed by an individual who had originally marked some, or all, of the items subject to review. The figure of 11,632 represents approximately 9.3% of all review applications reviewed that year.

34. The applications in question can be broken down as follows:

Type of review Qualification Number of affected reviews Number of affected reviews where a fee was charged (because no grade change)
Whole script A level 95 85
Whole script GCSE 78 59
Item Level A level 3,089 2,576
Item Level GCSE 8,370 6,816
Total for 2016   11,632 9,536

Total for 2016 - 2019 series

35. In total, Pearson conducted 519,151 reviews of marking in respect of its GCSE and A level qualifications over the 4 exam series in 2016, 2017, 2018 and 2019. Of those, 46,797 applications have been identified by Pearson as having been reviewed by an individual who had originally marked some, or all, of the items subject to review. Of those, 36,807 were reviews where a fee was charged to a Centre because there was no grade change made following the review. The figure of 46,797 represents approximately 9.0% of all review applications reviewed by Pearson between 2016 and 2019. 45,616 of these related to item level marking (approximately 97%) and 1,181 related to whole script level marking (approximately 3%).

Reviews of moderation

36. Pearson has confirmed that in all cases from 2016 to 2019, reviews of moderation were carried out by someone other than the original moderator. Pearson stated that it has compared the moderator identification number of the original moderator who was allocated to the centre with the moderator identification number of the review moderator who completed the review of moderation. There were no instances where the 2 coincided.

Root cause analysis

37. In the event notification submitted on 19 November 2019, Pearson reported that the systems and processes that it had in place for PRS reviews prevented an examiner from conducting a review of marking if that examiner had originally marked the whole script in its entirety. The same principle was applied for item-based scripts with 3 or fewer items. However, a small number of exceptions were made to this when the availability of other assessors was scarce and/or where the external service level agreement for completion of the review was at risk of being exceeded.

38. For item-based scripts with more than 3 items, no such control was in place. Pearson explained that this was because the chance of any individual examiner being the original marker of a substantial portion of the script was low. Pearson said that since the examiners selected to conduct reviews of marking are generally more experienced and are normally senior examiners, they are likely to have low original marking allocations, which further reduces the chance that the review marker of a particular script will have been involved in the original marking. In addition, the original item level marking is anonymised and cannot be linked to a centre or candidate number.

39. Pearson explained that when the Conditions relating to reviews of marking and moderation were updated (that is, when the GCE and GCSE Qualification-level Conditions and Requirements were published in 2016), changes were not made to its processes to ensure it remained fully compliant. Reviews of marking were therefore in some cases allocated to the examiner who had originally marked the script, in full or in part.

40. In its Summary of the Review Marking issues document dated 6 December 2019, Pearson provided information regarding its previous exception processes:

“Whole marked scripts were only allocated to the original examiner to review when it meant that completing the review to external deadlines was at risk. This pressure was exacerbated when there are low entry components that were originally sole marked. Such components are often in low entry qualifications for which the type of subject means that there is a limited pool of qualified examiners to draw upon. To mitigate the risk of the original examiner completing the review and to maintain compliance with the conditions, we instructed them to complete a report explaining why they were changing marks.

For item marked scripts with more than 3 questions, we believed that the anonymity that epen affords meant that we did not need to allocate scripts which had been part marked by a reviewing examiner to another reviewer. As the original examiner does not know the origin of the part of the script they are reviewing, they are not able to identify it should they see it again through reviews of marking. As such, we believed that we were compliant with Condition GCE/GCSE 17.6 (b).

For item marked scripts with 3 or fewer questions on the paper, in order to ensure compliance with GCE/GCSE 17.6 (b), we had a process in place that prevented these scripts from being allocated to a reviewing examiner who had originally marked one or more of the items on the scripts. There were a small number of exceptions to this when it meant that external deadlines were at risk. This pressure is exacerbated when there are low entry components that were originally sole marked. Such components are often in low entry qualifications for which the type of subject means that there is a limited pool of qualified examiners to draw upon.

For both whole marked scripts, and scripts marked by item with 3 or fewer items, when the script was allocated to a post results reviewer who had already marked some or all of the script, we logged this and required the reviewer to complete a report if they were proposing to change marks.”

41. In a letter dated 6 March 2020, Pearson further explained that:

“Our investigation into the root cause of the error has determined that we had been operating under the false assumption that our reviews of marking were compliant. The changes to the conditions in this regard represented a minor editorial change involving the removal of 2 words which had wide ranging implications for the successful and compliant provision of reviews of marking services. Also, we cannot recall, nor find any documented record of any discussion of this change from the Review of Marking Working Group or our records of our response to the consultation.”

42. Ofqual does not consider this to be an accurate reflection of the changes that were made to reviews of marking in 2016 and notes that Pearson is listed as responding to the consultation. Further, Ofqual does not consider that the controls Pearson put in place (requiring an examiner to complete a form if they proposed to change a mark) mitigated the risks that the Conditions seek to avoid (that is, that an examiner may consciously or otherwise, wrongly approve their own marking).

Previous correspondence with Ofqual in 2018

43. On 14 September 2018, Ofqual wrote to Pearson to highlight issues that had been identified during Ofqual’s review of summer 2017 appeals in order to ensure awareness of the possible risk of non-compliance in a number of areas of concern. One of the areas of concern was in relation to examiners on appeal panels having previously been involved in marking or reviews of marking. Among other things, Ofqual asked Pearson to confirm whether it had identified any panels where the same examiners were involved in marking and reviews of marking.

44. On 11 October 2018, Pearson responded to Ofqual and said:

“We can confirm that we are able to ensure that reviews of marking are carried out by examiners with no previous involvement in the original marking.”

45. On 19 December 2019, Ofqual wrote to Pearson regarding its initial consideration of the case and noted that it had identified an apparent contradiction between the statements made in Pearson’s letter dated 11 October 2018 and the explanation in the 6 December 2019 summary document. In particular, Ofqual noted that it was unable to readily reconcile the October 2018 statement that “we can confirm we are able to ensure that reviews of marking are carried out by examiners with no previous involvement in the original marking” with the account provided in December 2019 in which Pearson:

  1. (a) acknowledged that in a variety of circumstances it had knowingly allocated reviews to examiners with previous involvement in the original marking;
  2. (b) referred to the safeguards it had in place to mitigate the risk presented by this approach; and
  3. (c) asserted that until recently it considered its approach to be in compliance with the conditions.

46. In summary, it appeared from the 6 December 2019 document that Pearson was not able to ensure, or chose not to ensure, that reviews of marking were carried out by examiners with no previous involvement in the original marking, and that in fact reviews of marking had been conducted by original examiners on numerous occasions, in a variety of circumstances. Ofqual asked Pearson to explain the 2 apparently contradictory positions.

47. On 13 January 2020, Pearson explained that the October 2018 statement:

“…was made in the context of a much longer exchange specifically about appeals processes, which followed feedback from Ofqual on a sample of appeals from summer 2017. The comment was made in good faith and reflected [the author’s] understanding of the review of marking processes at the time. It was not until the appeal case referred to in our original event notification was reviewed in November 2019, and an internal investigation subsequently carried out, that the facts of the matter became clear.”

Evidence and findings in respect of failure to comply with Conditions of Recognition

48. Pearson is recognised as an awarding body by Ofqual under section 132(1) of the Apprenticeships, Skills, Children and Learning Act, 2009 and is subject to the General Conditions of Recognition which Ofqual is required to set and publish under Section 134 of the 2009 Act.

49. In addition to the General Conditions of Recognition, Pearson is subject to Qualification Level Conditions and Requirements in respect of the GCSE and A level (GCE) qualifications that it offers.

50. Pearson has a legal obligation to comply with its Conditions of Recognition on an ongoing basis and is required to have regard to the Guidance associated with the Conditions.

51. Further information about Ofqual’s statutory powers and the Conditions of Recognition are set out at Annex A (Legal Provisions).

GCE / GCSE Qualification Level Conditions

52. The relevant Qualification Level Conditions that were in force during the 2019 series were:

  1. (a) GCE Qualification Level Conditions and Requirements (April 2019); and
  2. (b) GCSE (9-1) Qualification Level Conditions and Requirements (April 2019).

53. There were a variety of different Qualification Level Conditions in force during the 2016, 2017 and 2018 series depending on whether the affected qualifications were legacy or reformed qualifications. For the purposes of this case, there is no material difference between the relevant Conditions across 2016 - 2019. For ease of reference, the 2019 Conditions will be referred to in this document and will be referred to collectively as ‘GCE / GCSE’ Conditions.

GCE / GCSE Condition 17

54. GCE / GCSE Condition 17.1 provides that:

“In respect of each GCE / GCSE Qualification which it makes available an awarding organisation must establish, maintain and comply with arrangements for it to carry out a review of marking of a Learner’s Marked Assessment Material.”

55. GCE / GCSE Condition 17.6 states that:

The arrangements must provide that –

(a) all reviews of marking will be carried out by Assessors who have appropriate competence and who have no personal interest in the outcome of the review being carried out,

(b) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking in respect of that task.

56. The guidance to the GCE / GCSE Conditions states that a ‘personal interest’ is a conflict of interest that relates to a particular individual and falls within the definition under General Condition A4.1(b) and (c). The relevant question to ask is whether the person carrying out the review has any reason to make anything other than a decision made in good faith in line with the relevant conditions, or whether an informed and reasonable observer would conclude that such a reason exists.

Evidence of breach

57. Ofqual considers that there is evidence to indicate that Pearson failed to comply with GCE / GCSE Condition 17.6(b) in respect of:

  1. (a) the affected 45,616 item level reviews in 2016, 2017, 2018 and 2019. This is on the basis that Pearson has reported to Ofqual that items in those assessments were reviewed by the same person who had originally marked those items.

  2. (b) the affected 1,181 whole script reviews in 2016, 2017, 2018 and 2019. This is on the basis that Pearson has reported to Ofqual that the entire scripts were reviewed by the same person who had conducted the original marking.

58. Ofqual considers that there is evidence to indicate that Pearson failed to comply with GCS / GCSE 17.6(a) in respect of:

  1. (a) the affected 1,181 whole script reviews in 2016, 2017, 2018 and 2019. This is on the basis that due to the nature of the whole script marking method, there must have been an increased likelihood that the reviewer may have recognised the material that they were reviewing. There was therefore the possibility that an informed and reasonable observer may conclude that the person carrying out the review could have reason to seek to minimise changes to their own original marking. Ofqual acknowledges that there is no evidence to suggest that the reviewers in fact did so in this case, or that that they did not have the appropriate competence to carry out the review.

59. Ofqual has not alleged a breach of GCE / GCSE Condition 17.6(a) in respect of the 45,616 affected item level reviews in 2016, 2017, 2018 and 2019 because it appears that this method of marking provided sufficient anonymity to the extent that it is unlikely that the reviewer would have been able to recognise their own original marking. This is on the basis that Pearson has reported to Ofqual that the original item level marking is anonymised and cannot be linked to a centre or candidate number. As the original examiner does not know the origin of the part of the script they are reviewing, they are not able to identify it should they see it again through reviews of marking. The only potential exception to this is item level reviews that were originally sole marked by the Principal Examiner in low entry qualifications. In these instances, there may have been an increased risk of the reviewer being able to identify that they had conducted the original marking which may give rise to them having a personal interest in the outcome of the review. However, Ofqual considers that these instances would likely have been very small in number and as there is no evidence to suggest any prejudice has been caused to any Learners as a result, Ofqual does not consider it is either necessary or expedient to seek to establish the extent of these occurrences for the purposes of alleging a breach of 17.6(a) in this narrow respect. Pearson has also provided assurance to Ofqual through its undertaking and Action Plan that it intends to recruit additional reviewers for exams which have previously been small entry or sole marked papers to ensure compliance going forward.

Summary of Pearson’s response

60. Pearson accepts that it was in breach of Condition GCE/GCSE 17.6(a) in respect of 1,181 whole script reviews because there was a possibility that an examiner may recognise a whole script as one they had previously marked, and in such an event this could constitute the examiner having a ‘personal interest’ in the outcome of the review of marking process. In such cases, the individual reviewing their own marking was required [if they were proposing to change marks] to write a report on the outcome of their review to help mitigate this risk.

61. Pearson accepts that it was in breach of Condition GCE/GCSE 17.6(b) in respect of 45,616 item level reviews and 1,181 whole script reviews because its processes did not ensure that an assessor who was previously involved in the marking of a task in an assessment in respect of a learner must not be involved in a review of marking in respect of that task.

Enforcement Committee’s finding

62. The Enforcement Committee considered the evidence and Pearson’s admissions in respect of these alleged breaches and found that the breaches were proven for the reasons set out above.

General Conditions of Recognition

A5 – Availability of adequate resources and arrangements

63. General Condition A5.1(a) requires that an awarding organisation must ensure that it has the capacity to undertake the development, delivery and award of qualifications which it makes available, or proposes to make available, in a way that complies with its Conditions of Recognition.

64. For the purposes of Condition A5.1, Condition A5.2(a) requires that an awarding organisation must establish and maintain arrangements which will ensure that it retains at all times a Workforce of appropriate size and competence.

Evidence of breach

65. Ofqual considers that there is evidence to indicate that Pearson failed to comply with Condition A5.2(a). This is on the basis that Pearson reported to Ofqual in its event notification on 19 November 2019, that in relation to whole script marking and item level marking with 3 or fewer items, it had systems and processes in place to prevent an examiner from conducting a review of their own marking. However, an exceptions process was applied to this “when the availability of assessors was scarce and/or where the external service level agreement for completion of the review was at risk of being exceeded.”

66. Further, in its summary document dated 6 December 2019, Pearson reported that this exceptions process applied “when it meant that completing the review to external deadlines was at risk. This pressure was exacerbated when there are low entry components that were originally sole marked. Such components are often in low entry qualifications for which the type of subject means that there is a limited pool of qualified examiners to draw upon.”

67. This demonstrates that Pearson did not retain sufficient numbers of examiners, particularly in low entry components, to enable it to carry out its reviews of marking in a way that was compliant with the Conditions (that is, it did not have sufficient resource to allocate reviews of marking to an examiner that had not conducted any of the original marking). Therefore, Pearson cannot be said to have retained a Workforce of appropriate size.

Summary of Pearson’s response

68. Pearson accepts that it was in breach of Condition A5.2(a) because in some instances it did not have sufficient resource in place to allocate reviews of marking to an examiner that had not conducted any of the original marking. Pearson noted that because it helps to ensure that there is a wide range of GCSE and A level qualifications available, some subjects such as in languages, have limited registrations each year and there is a limited pool of subject-expert examiners for it to draw upon. This is an area that is being addressed within Pearson’s action plan.

Enforcement Committee’s finding

69. The Enforcement Committee considered the evidence and Pearson’s admissions in respect of these alleged breaches and found that the breaches were proven for the reasons set out above.

Evidence of breach

70. Condition A5.2(e) requires that an awarding organisation must establish and maintain appropriate systems of planning and internal control.

71. Ofqual considers that there is evidence to indicate that Pearson failed to comply with Condition A5.2(e). This is on the basis that Pearson reported to Ofqual in its event notification on 19 November 2019 that when the Conditions relating to reviews of marking were updated in 2016, “changes were not made to its processes to ensure it remained fully compliant”. And in its letter dated 6 March 2020, Pearson acknowledged that it “had been operating under the false assumption that our reviews of marking were compliant” and goes on to demonstrate a fundamental lack of knowledge or understanding of the changes that were consulted upon and then implemented in 2016.

72. Further, in its summary report dated 6 December 2019, Pearson reported that when it allocated reviews of marking to an examiner that had conducted all or part of the original review, it had a process in place to mitigate the risk. The process required the examiner to complete a report if they were proposing to change marks. In a letter dated 6 March 2020, Pearson confirmed that it did not routinely ask reviewers who had marked some or all of the original scripts the reasons for any outcomes which did not result in a mark change. Ofqual considers that this process failed to properly identify and mitigate the risk that the GCE / GCSE Conditions seek to avoid, which is that an examiner who is asked to review their own marks, may be either incentivised to minimise changes to their original mark, or may be subject to confirmation bias and fail to spot their own errors. Requiring a report explaining only when changes are proposed, rather than when changes are not made, only serves to aggravate these risks and does nothing to mitigate it.

73. Therefore Pearson cannot be said to have had appropriate systems of planning and internal control in place.

Summary of Pearson’s response

74. Pearson accepts that it was in breach of Condition A5.2(e) for the reasons set out in Ofqual’s summary of the evidence above.

Enforcement Committee’s finding

75. The Enforcement Committee considered the evidence and Pearson’s admissions in respect of these alleged breaches and found that the breaches were proven for the reasons set out above.

A6.1 – Identification and management of risks

76. Condition A6.1 requires that an awarding organisation must take all reasonable steps to identify the risk of the occurrence of any incident which could have an Adverse Effect. An Adverse Effect is defined as an act, omission, event, incident, or circumstance which gives rise to prejudice to Learners or potential Learners, or adversely affects (i) the ability of the awarding organisation to undertake the development, delivery or award of qualifications in a way that complies with its Conditions of Recognition, (ii) the standards of qualifications which the awarding organisation makes available or proposes to make available, or (iii) public confidence in qualifications.

Evidence of breach

77. Ofqual considers that there is evidence to indicate that Pearson failed to comply with Condition A6.1. This is on the basis that Pearson failed to act upon information that was flagged to it by Ofqual in September 2018 regarding the risks of non-compliance with the Conditions where examiners are involved in marking and reviews of marking and/or preliminary appeal reviews of scripts. Pearson was asked to consider the issue and take all reasonable steps to prevent such an occurrence. In its response on 11 October 2018, Pearson purported to provide assurance to Ofqual that it was able to ensure that reviews of marking were carried out by examiners with no previous involvement in the original marking. Subsequent correspondence from Pearson in relation to this matter shows that this was not the case, or that even if it was able to do this it chose not to do so, and that Pearson had in fact been allocating reviews of marking to examiners who had been involved in the original marking, since 2016.

78. Therefore Pearson cannot be said to have taken all reasonable steps to identify the risk of this incident occurring, and it was an incident which Ofqual considers adversely effected Pearson’s ability to comply with its GCE / GCSE Conditions.

Summary of Pearson’s response

79. Pearson does not accept that it breached Condition A6.1. Pearson considers that Ofqual flagged an issue that was specific to ‘appeal panels’ only. Pearson confirmed its compliance with this area and has continued to remain compliant. However, Pearson’s response to Ofqual inadvertently and unintentionally mentioned Reviews of Marking in the introduction to its response, despite the Ofqual query being concerned with the appeal process that follows the review of marking stage. This was unfortunate but was intended to be in response to the specific question asked about the appeal process, and the reference to reviews of marking was included based on the belief that its processes for reviews of marking were also compliant. Pearson considers that it was evident from the Ofqual question, and the detail of its response, that the risk being raised by Ofqual concerned the appeals process, and it was specifically in relation to the appeals process where Pearson was intended to provide assurance about its compliance. In that context, Pearson’s intention was to comply and it believed it had taken all reasonable steps to identify the risk of the occurrence of any incident which could have an Adverse Effect.

Enforcement Committee’s finding

80. The Enforcement Committee considered the evidence and Pearson’s representations in respect of this alleged breach and found that the alleged breach of Condition A6.1 was proven. The Enforcement Committee found that the wording in Ofqual’s question and Pearson’s response was clear that it related to reviews of marking and therefore it was reasonable to expect that this correspondence should have prompted Pearson to identify that it was not conducting its reviews of marking in a way that was compliant with its conditions of recognition. The fact that it failed to do this meant that opportunities were missed to identify and remedy the problem for the 2019 series. Consequently, Pearson cannot be said to have taken all reasonable steps to identify the risk of the occurrence of an incident which could have an Adverse Effect, in breach of Condition A6.1.

Impact of failure to comply

Adverse Effects

81. Pearson has accepted that over a 4 year period, 46,797 reviews of marking were not conducted by a fresh examiner as required by its Conditions of Recognition.

82. Pearson has confirmed that nearly all (around 99%) of the affected reviews were conducted by its most senior group of examiners.

83. Pearson has conducted a repeat review of marking for a statistically significant sample (2,371) of the affected cases in 2019 (data was not available to conduct a similar review for earlier years). Of those, 2,360 grades were found to be accurate, giving a grade-agreement rate of 99.5%. This exceeds the total agreement-rate in a normal PRS window. Ofqual is therefore satisfied that there appears to be no evidence to suggest that the original review of marking service offered was materially affected by the non-compliance.

84. While there appears to be no evidence that Pearson’s failure to comply with the Conditions had any material impact on the outcome of any review or caused any prejudice to Learners, the Enforcement Committee considered that the requirements in GCE / GCSE Conditions 17 are an integral part of the assessment process and fundamental to securing high quality assessments, standards and public confidence in GCSE and A level qualifications. A failure to provide an independent review of marking, particularly across a large proportion of cases over a significant period of time, is therefore likely to seriously undermine public confidence in the review and appeals system and the qualifications system more generally.

Income from affected reviews

85. Pearson charges a fee to Centres for conducting its review of marking service. In total, Pearson conducted 519,151 reviews of marking in respect of its GCSE and A level qualifications over the 4 exam years in 2016, 2017, 2018 and 2019. Of those, 46,797 applications have been identified by Pearson as having been reviewed by an individual who had originally marked some, or all, of the items subject to review. Of those, 36,807 were reviews where a fee was charged to a Centre because there was no grade change made following the review.

86. Pearson has committed (through an undertaking provided to Ofqual) to reimbursing Centres that it charged for non-compliant reviews in 2016, 2017, 2018 and 2019. Pearson has calculated the amount to be reimbursed to Centres in a way which is proportionate to the extent of the non-compliance. For example, if a review of a script was conducted by an examiner who had marked some of the original script, the amount refunded will be proportionate to the volume of items the review marker had already marked. If a script was whole marked, then the whole fee would be refunded.

87. Pearson has calculated the total sum to be reimbursed to Centres to be £320,510 which is made up as follows:

  1. (a) Item level marking - £274,796
  2. (b) Whole script marking - £45,714

88. Pearson has confirmed that the total number of Centres to be reimbursed is 3,970, broken down by country as follows:

  1. (a) England – 3,858
  2. (b) Wales – 64
  3. (c) Northern Ireland – 48

89. The total number of learners affected at the Centres was 32, 241, broken down by country as follows:

  1. (a) England – 31,978
  2. (b) Wales – 142
  3. (c) Northern Ireland 121

Avoided costs

90. Pearson has confirmed that with the change to the Conditions in 2016, it would have needed to implement a similar short-term solution as was outlined in its action plan for 2020 and for which the costs of £298,000 were estimated. This would have been in lieu of a more permanent systems-based solution to be in place in time for the 2016 post results service window in order to ensure compliance. However, Pearson would then have looked to ensure the long-term solution outlined in its action plan was implemented in 2017.

91. This permanent systems-based solution of item level reviews of marking would ensure compliance for reviews of marking as reviewing examiners would only be able to view and mark items which they had not previously had any involvement in assessing. With this approach, Pearson would not incur the additional costs that comprise the £298,000 total quoted for the short-term solution as the additional work of both processing and assessment staff would not be required for reviewing and allocating cases as the system would do this automatically. Furthermore, no additional fees would have been paid for each review which has to be repeated to ensure compliance as each script reviewed would effectively be broken down into its constituent parts at item level and then each item would only need to be marked once by eligible examiners, therefore ensuring no repeat review is required.

92. Pearson therefore estimated the additional cost that would have been incurred in 2017 to deliver this solution would have been £125,000. This £125,000 is all on system development work and breaks down with £50,000 being attributed to development to the system responsible for allocations and then £75,000 for development work to the marking platform itself.

93. Due to re-prioritisation of resources in technology required to deliver the teacher assessed grades process for summer 2021, Pearson has been unable to finish the system development work required for this item level solution and therefore does not know what the actual costs of delivery will be. It is therefore not possible to know exactly what the actual cost incurred in 2017 would have been. With regards to 2018 and 2019 costs, Pearson would not have expected to incur any additional costs in ensuring compliance in 2018 or 2019 over what was already incurred in completing the reviews of marking, with the long-term system-based solution in place.

94. In summary, Pearson would have expected to incur a total of £423,000 in additional costs, broken down for each year between 2016 and 2019 as follows:

  1. (a) 2016 - £298,000
  2. (b) 2017 - £125,000
  3. (c) 2018 - £0
  4. (d) 2019 - £0

95. Pearson noted that these costs have been delayed but will still be incurred by Pearson when implementing the short and long-term solution set out in its action plan.

96. In relation to the costs that would have been incurred had Pearson recruited and retained the necessary additional over the course of several years Pearson confirmed in a letter dated 18 March 2022 that:

…the need to have recruited additional examiners in order to ensure compliance with the GCE/GCSE conditions would only have applied in a small number of cases for subjects with particularly small entries which were sole marked by a senior examiner and where no other examiner was recruited. For the majority of non-compliant cases there would have been additional ROMM examiners already recruited for the subject meaning that ROMM work should have been allocated to them, ensuring compliance. This would not have resulted in any additional cost, as the ROMM fee would have been paid the to them instead of to those examiners who were incorrectly allocated the work. There would have been no additional training costs as these examiners had already been trained to carry out ROMMs.

We have analysed our review of marking data from 2016 to 2019 to confirm the number of cases where additional examiners would have needed to be recruited. Of the 1,352 papers between these years where scripts were reviewed and the review did not comply with the conditions, 34 of these papers were sole marked and would have required additional examiners to be recruited. The cost associated with training these additional examiners would be low, as they would be the Principal Examiners from other papers in the same qualification and already familiar with the other papers in the qualification. The cost for RoMM training for these examiners is determined by the script fee and the overall costs are in the table below:

Year No. papers Total cost
2016 14 £240
2017 5 £110
2018 10 £230
2019 5 £50
Total 34 £630

Enforcement Committee’s finding

97. The Enforcement Committee noted that it was difficult to make a determination of fact regarding the specific amount of costs that Pearson has avoided incurring as a result of its non-compliance with its Conditions of Recognition, in particular the extent to which Pearson avoided costs year on year in circumstances in which the majority of the costs are said to be one-off.

98. The Enforcement Committee therefore took a holistic view and considered that any penalty imposed should include both a deterrent and restitutive element to reflect the fact that Pearson has avoided some costs by being non-compliant with its Conditions of Recognition for a prolonged period of time, noting however that Pearson would still incur costs in the future.

Undertaking

99. On 8 June 2020, Pearson provided an undertaking[footnote 17] (dated 8 June 2020) to Ofqual in which it admitted a failure to comply with its Conditions of Recognition in respect of 46,797 reviews of marking in 2016, 2017, 2018 and 2019. Pearson charged a fee to Centres in respect of 36,807 of those reviews because there was no grade change following the review.

100. Pearson agreed in the undertaking that it will compensate affected Centres in the form of credit notes to the value of £320,510. This sum has been calculated in a way that is proportionate to the extent of non-compliance. The payments to Centres will be banded to ensure the refund figure is meaningful.

101. Pearson has also undertaken to comply with an action plan to ensure compliance with its Conditions of Recognition in the future. Pearson provided an updated action plan in August 2021[footnote 18]

102. On 18 March 2022, Pearson provided an update on its performance against the action plan. In this update, Pearson confirmed that:

…the proposed short-term actions were implemented and we are undertaking reviews of marking in compliance with Ofqual’s conditions.

We can confirm that we have a long-term solution in place and which will be in effect for the next summer exam series in 2022. Since we last communicated with Ofqual on this matter on 4 August 2021, we have been working with our development and delivery teams and based on the evaluation of the options available our final approach, whilst fully compliant, differs from the early detail we shared with you. Whereas the previous long-term solution was based on an entirely new process utilising our live marking platform to complete reviews of marking at item level, we have since confirmed it possible to build upon existing processes with some additional system development. By building on existing processes it also gives us greater confidence in terms of delivery for the summer 2022 series as opposed to implementing a brand new process for the first time. This is an extension to the ‘short-term’ solution that we implemented to reviews of marking in 2020 and 2021 ensuring that we are able to deliver at scale.

With regards to having this ‘long-term’ solution in place for our non-regulated January 2022 series, we can confirm that the required system development has taken place and the new functionality has been tested successfully and is being utilised for reviews of marking for that January series. There are some final, minor, developments to be completed to our internal processes and systems to achieve greater efficiency when dealing with volume in advance of the summer 2022 series.

103. Ofqual is assured by this that Pearson’s arrangements for reviews of marking for the autumn 2021 series were compliant with the Conditions and that is has arrangements in place to ensure compliance for future exam series.

Determination of a Monetary Penalty

104. On 26 April 2022, Ofqual’s Enforcement Committee considered the evidence and the partial admissions made by Pearson and found that Pearson has breached the following Conditions of Recognition in relation to its reviews of marking in 2016, 2017, 2018 and 2019, for the reasons set out above.

  1. (a) GCE / GCSE Condition 17.6 (a) and (b)
  2. (b) General Condition A5.2(a)
  3. (c) General Condition A5.2(e)
  4. (d) General Condition A6.1

105. In determining whether or not a Monetary Penalty is an appropriate regulatory outcome in this case, and if so, what amount would be proportionate to impose, the Enforcement Committee had regard to Ofqual’s Taking Regulatory Action policy (2012) and, in particular, the following aggravating and mitigating factors:

Aggravating factors

  1. (a) Pearson has made serious and persistent failings in the service it offers to its Centres and Learners by failing to provide a review of marking service that was compliant with its Conditions of Recognition;

  2. (b) These were extensive breaches which occurred on a wide scale affecting around 47,000 reviews of marking;

  3. (c) The breaches were prolonged and repeated over a 4 year period;

  4. (d) These were serious breaches of Conditions that are integral to the effectiveness and purpose of the system of reviewing marking. The failures have the potential to seriously undermine public confidence in the review of marking system and the qualifications system more generally;

  5. (e) The circumstances of the breaches were substantially within the control of Pearson. It had been knowingly allocating reviews to examiners with previous involvement in the original marking and it did not put adequate controls in place to guard against the risks the Conditions seek to avoid;

  6. (f) Pearson said that it had been operating under the misapprehension that it was compliant with its Conditions of Recognition and was unaware of the ‘zero-tolerance’ approach to the interpretation of GCE/GCSE Condition 17.6 with regards to item level marking and small entry subjects, even where it had considered that controls were in place to mitigate the risk of personal involvement in the original marking. This gives rise to concerns about Pearson’s management and control systems. Ofqual considers that Pearson should have been aware of Ofqual’s approach to, and interpretation of, the GCE/GCSE Conditions which were consulted on and also notes that the Conditions do not permit any mitigations or exceptions to be made, even for small entry subjects;

  7. (g) Pearson did not identify the risk of non-compliance, even when the issue was specifically drawn to its attention by Ofqual in 2018. Opportunities to identify and remedy the problem were missed, resulting in a further year of non-compliance in 2019 with 11,645 affected reviews that could have been avoided;

  8. (h) Pearson did not identify the failings until a review of marking issue, which was discovered during a preliminary appeal investigation in November 2019, was viewed in light of regulatory action that Ofqual had taken against other awarding organisations for similar issues;

  9. (i) The length of time taken to identify the non-compliance means that records had been destroyed and people had left posts, so it was not possible to conduct a review of impact on earlier years and potentially correct grades;

  10. (j) Pearson avoided incurring significant costs by not changing its processes and not recruiting and retaining additional examiners over several years in order to secure compliance with the change in Conditions in 2016 (although it is noted that Pearson will still incur system development costs when implementing its action plan going forward).

Mitigating factors

  1. (a) Pearson has given an undertaking to comply with an action plan to secure future compliance;

  2. (b) Pearson has given an undertaking to compensate Centres in relation to non-compliant reviews totalling £320,510.

  3. (c) Nearly all affected reviews (around 99%) were conducted by the most senior examiners who had received training. Their reviews were monitored and sampled during the review process to ensure marking standards were maintained;

  4. (d) It appears from the sampling exercise conducted by Pearson that there is no evidence that the non-compliance had a material impact on the outcome of the review or caused any prejudice to Learners;

106. The Enforcement Committee also considered:

  1. (a) the fact that the affected breaches amounted to approximately 9% of the total reviews conducted by Pearson between 2016-2019 and noted that this was a relatively high amount. It did not however consider this to be an aggravating or mitigating feature of the case;

  2. (b) any financial gain that Pearson made from charging fees for services that it had not adequately delivered (taking into account the offer to pay compensation to Centres);

  3. (c) any financial benefit that Pearson accrued in comparison to other awarding organisations, for failing to put in place adequate systems of planning and internal control and a Workforce of appropriate size and competence (see ‘Avoided Costs’ section of this Notice);

  4. (d) the need to deter Pearson and other awarding organisations from making similar failings in the future;

  5. (e) the need to promote public confidence in qualifications through visible, appropriate and effective regulatory action;

  6. (f) Pearson’s compliance history (it received a £85,000 fine in 2016);

  7. (g) the nature and circumstances of these breaches in comparison to other similar breaches for which fines have been imposed by Ofqual on other Awarding Organisations (see ‘Relevant Cases’ section of this Notice);

  8. (h) Pearson’s relative size and turnover as an awarding organisation.

Relevant cases

107. The Enforcement Committee considered the circumstances of this case in comparison to other relevant cases in which Ofqual has taken regulatory action. It considered that the most relevant case was the Monetary Penalty imposed on AQA in January 2020.

108. This was a decision to accept a Settlement Proposal from AQA to pay a fine to Ofqual of £350,000 in addition to voluntary payment of compensation of £735,750 to Centres in relation to similar failures in its reviews of marking arrangements.

109. The Enforcement Committee considered that the Monetary Penalty in this case should be higher than that imposed in the AQA case for the following combination of reasons:

  1. (a) This case is factually more serious than the AQA case because:

    1. (i) The failings spanned a 4 year period (as opposed to 3 years in the AQA case);

    2. (ii) The non-compliance occurred as a result of systemic issues with Pearson’s risk management and systems of planning and internal control in circumstances in which Pearson failed to take steps to alter its processes when the changes to the Conditions of Recognition were introduced;

    3. (iii) Pearson had been knowingly allocating reviews to examiners with previous involvement in the original marking and failed to put adequate controls in place to guard against the risks that the Conditions seek to avoid

    4. (v) Pearson missed an opportunity to rectify the issue in 2018 when contacted by Ofqual about the issue, leading to a further year of breaches which could have been avoided.

  1. (b) While Pearson made partial admissions to the alleged breaches, it did not at any stage during the enforcement process take up the opportunity to enter into settlement discussions with Ofqual or make any form of settlement proposal. Pearson is therefore not entitled to any settlement discount (a significant discount was afforded to AQA to reflect the settlement proposal that was made).

  2. (c) While Pearson has undertaken to pay compensation to Centres, the figure proposed by Pearson of £320,510 only relates to income it received from affected reviews of marking and does not include a figure to reflect the avoided costs that it should have incurred had it been compliant. (The voluntary compensation figure paid by AQA to Centres included income from affected reviews and its estimated avoided compliance costs).

  3. (d) Pearson’s size and turnover is significantly greater than that of AQA’s and AQA is an education charity operating for the public benefit, whose surpluses are reinvested into loss making qualifications and the provision of CPD).

Decision

110. Taking all of the above into account, the Enforcement Committee has decided to give notice of its intention to impose a Monetary Penalty on Pearson in the sum of £1,200,000.

111. The Enforcement Committee is satisfied, in accordance with section 151B of the Apprenticeships, Skills, Children and Learning Act 2009, that a Monetary Penalty in this sum would not exceed 10% of Pearson’s total annual turnover.

Representations

112. This Notice has been served on Pearson in accordance with section 151A(4) of the Apprenticeships, Skills, Children and Learning Act, 2009 and Pearson now has the opportunity to make representations in respect of Ofqual’s intention to impose a Monetary Penalty of £1,200,000 on it. Any such representations must be made in writing and sent by email to [email protected] and must be received before 4pm on 25 July 2022.

113. Ofqual will consult with Pearson regarding a proposed date for publication of this Notice on its website and the Portal. Once this Notice has been published, interested parties will also have the opportunity to make representations in respect of Ofqual’s intention to impose a Monetary Penalty on Pearson, in accordance with Ofqual’s Taking Regulatory Action policy (2012).

Next Steps

114. The Enforcement Committee will consider this case again once Pearson and interested parties have had the opportunity to make representations.

115. The Enforcement Committee will consider any representations made as specified in this Notice (Representations) and will decide whether to make a final order for the payment of a Monetary Penalty, and if so in what amount, or whether any other order should be made.

116. If Ofqual does not receive representations it may determine this matter after the date for representations is given, alternatively it may agree a different date for the receipt of representations.

Statutory powers

1. Pearson Education Limited (“Pearson”) is recognised as an awarding body by The Office of Qualifications and Examinations Regulation (“Ofqual”) under section 132(1) of the Apprenticeships, Skills, Children and Learning Act, 2009 (“the 2009 Act”) and is subject to the General Conditions of Recognition (“the Conditions”) which Ofqual is required to set and publish under Section 134 of the 2009 Act.

2. Under Section 151A(2) of the 2009 Act, Ofqual may impose a Monetary Penalty on an awarding body if it appears to Ofqual that the awarding body has failed to comply with its Conditions of Recognition.

3. Under Section 151B(3) of the 2009 Act, the amount of any Monetary Penalty may be whatever Ofqual decides is appropriate in all the circumstances of the case, subject to Section 151B(1), which provides that Ofqual may not impose a Monetary Penalty in an amount which exceeds 10% of the awarding body’s turnover.

4. Ofqual’s Taking Regulatory Action Policy (2012)[footnote 19] sets out how it will use its powers to take regulatory action, including the factors it will take into account when deciding whether to impose a Monetary Penalty and how it will determine the amount of any Monetary Penalty to be imposed.

General Conditions of Recognition

5. Pearson has a legal obligation to comply with the General Conditions of Recognition on an ongoing basis. The General Conditions of Recognition include guidance on how to comply with the rules. Pearson has a legal obligation to have regard to this guidance.

6. The relevant General Conditions of Recognition in this case are:

  1. (a) A5.1 (as exemplified by A5.2(a) and (e)
  2. (b) A6.1

7. Condition A5.1 provides that:

“An awarding organisation must –

  1. (a) ensure that it has the capacity to undertake the development, delivery and award of qualifications which it makes available, or proposes to make available, in accordance with its Conditions of Recognition, and
  2. (b) take all reasonable steps to ensure that it undertakes the development, delivery and award of those qualifications efficiently.”

8. Condition A5.2 provides that:

“For these purposes, an awarding organisation must establish and maintain –

  1. (a) arrangements which will ensure that it retains at all times a Workforce of appropriate size and competence
  2. (b) …
  3. (c)…
  4. (d)…
  5. (e) appropriate systems of planning and internal control.”

9. The associated guidance to Condition A5 provides examples of positive indicators that would suggest an awarding organisation is likely to comply with this Condition. In particular, the awarding organisation:

• identifies and has in place, the resources it needs to develop, deliver and award its qualifications;
• 8 acts quickly to identify and address any shortcomings in its capacity or ability to develop, deliver or award any of its qualifications that it could not reasonably have foreseen.

10. The guidance also provides examples of negative indicators that would suggest an awarding organisation is not likely to comply with this Condition. In particular, the awarding organisation:

• does not identify or address inefficiencies in the development, delivery and award of its qualifications;
• fails to make appropriate amendments to the size and competence of its Workforce when it makes significant changes to the qualifications it offers.
• relies on IT systems that are prone to poor performance and/or repeated error.

11. Condition A6.1 provides that:

“An awarding organisation must take all reasonable steps to identify the risk of the occurrence of any incident which could have an Adverse Effect.”

12. The associated guidance for Condition A6 provides examples of positive indicators that would suggest an awarding organisation is likely to comply with this Condition. In particular, the awarding organisation:

• identifies events that might have an Adverse Effect using risk management approaches;
• knows where ownership for its approach to risk management lies within the organisation;
• reviews and updates its risks using a systematic and consistent approach.

13. The guidance also provides examples of negative indicators that would suggest an awarding organisation is not likely to comply with this Condition. In particular, the awarding organisation:

• does not systematically consider the range and type of risks that may have an impact on its regulated activities;
• fails to identify a foreseeable risk where it might be reasonably expected to do so that could result in an Adverse Effect.

14. Condition J1.8 defines an Adverse Effect as:

“An act, omission, event, incident, or circumstance has an Adverse Effect if it –

• gives rise to prejudice to Learners or potential Learners, or

  1. (b) adversely affects –

    1. (i) the ability of the awarding organisation to undertake the development, delivery or award of qualifications in accordance with its Conditions of Recognition,

    2. (ii) the standards of qualifications which the awarding organisation makes available or proposes to make available, or

    3. (iii) public confidence in qualifications.”

Qualification Level Conditions

15. In addition to the General Conditions of Recognition, Pearson is subject to Qualification Level Conditions and Requirements in respect of the GCE and GCSE qualifications that it offers.

16. The relevant Conditions in force during the 2019 series were:

• GCE Qualification Level Conditions and Requirements (April 2019); and • GCSE (9-1) Qualification Level Conditions and Requirements (April 2019).

17. There were a variety of different Conditions in force during the 2016, 2017 and 2018 series depending on whether the affected qualifications were legacy or reformed qualifications. For the purposes of this case, there is no material difference between the relevant Conditions across 2016 - 2019. For ease of reference, these Conditions are referred to collectively in this document as ‘GCE / GCSE’ Conditions.

18. The relevant Qualification Level Conditions in this case are:

• GCE / GCSE17 - Review of marking of Marked Assessment Material.

19. Condition GCE / GCSE 17.1 provides that:

In respect of each GCE / GCSE Qualification which it makes available, an awarding organisation must establish, maintain and comply with arrangements for it to carry out a review of marking of a Learner’s Marked Assessment Material.

20. Condition GCE / GCSE 17.6 provides that:

The arrangements must provide that –

  1. (a) all reviews of marking will be carried out by Assessors who have appropriate competence and who have no personal interest in the outcome of the review being carried out,

  2. (b) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking in respect of that task,

  3. (c) …

21. The guidance to the GCE / GCSE Conditions states that a ‘personal interest’ is a conflict of interest that relates to a particular individual and falls within the definition under Condition A4.1(b) and (c). The relevant question to ask is whether the person carrying out the review has any reason to make anything other than a decision made in good faith in line with the relevant Conditions, or whether an informed and reasonable observer would conclude that such a reason exists.

Annex B: Undertaking and Action Plan (June 2020)

Undertaking given by Pearson Education Ltd (‘Pearson’) to the Office of Qualifications and Examinations Regulation (‘Ofqual’), in accordance with B8 of the General Conditions of Recognition (The ‘Conditions’)

Failure to comply with Conditions

1. Pearson admits that in respect of its GCSE and GCE qualifications in 2016, 2017, 2018 and 2019, it failed to ensure that its arrangements for reviews of marking complied with GCE / GCSE Condition 17.6(b)[footnote 20], which provides that:

…an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking in respect of that task…

2. This failure to comply with the Conditions affected a total of 46,797 reviews of marking.

3. Pearson charges Centres a fee for its review of marking service, if the review does not result in a grade change. The number of affected reviews where a fee was charged was 36,807. The fees associated with those reviews totalled £320,510.

Undertaking

4. Pearson hereby undertakes to:

  1. (a) compensate all Centres with learners whose reviews of marking were not undertaken by a fresh examiner in 2016, 2017, 2018 and 2019, where a fee was charged. This compensation will be calculated based on the fees applicable at the time for the particular services requested and performed by Pearson and will be proportionate to the extent of non-compliance.

  2. (b) pay this compensation to affected Centres in the form of credit notes, on a date to be agreed with Ofqual. The credit notes will be redeemable against the full range of Pearson’s services, with choices being entirely in the hands of Centres. Pearson will ensure that the accompanying communications do not convey any promotional content or seek to advantage Pearson in any way.

  3. (c) perform the actions set out in Annex 1[footnote 21] within the timeframe set out therein in order to ensure compliance with the Conditions for the next exam series and beyond.

Declaration

5. In giving this Undertaking, I acknowledge on behalf of Pearson that:

  1. (a) Ofqual will publish this Undertaking (on a date to be agreed with Pearson); and

  2. (b) Ofqual may take regulatory action in respect of any failure to comply with the terms of this Undertaking which may include the issue of a Direction, the imposition of a Monetary Penalty, Costs, or any other such action as it deems appropriate; and

  3. (c) Ofqual may take other regulatory action in respect of the breaches set out in this Undertaking and any other breaches that may be determined by Ofqual’s Enforcement Committee in due course. Any regulatory action will be taken in accordance with Ofqual’s ‘Taking Regulatory Action’ policy (2012).

Timing

6. Ofqual and Pearson have agreed that:

  1. (a) Pearson’s payment of compensation to Centres (as set out in paragraph 4(a) of this Undertaking); and

  2. (b) Ofqual’s consideration of any regulatory action in respect of the breaches set out in this Undertaking (as set out at paragraph 5(c) of this Undertaking)

will be postponed until further notice to enable both parties to prioritise critical work associated with the Coronavirus pandemic.

Name: Derek Richardson

Responsible Officer, Pearson

Date: 8 June 2020

Pearson Action Plan for PRS compliance

Action plan

We outline here our high level action plan, with associated estimated timelines, to achieve compliance with the condition, GCSE/GCE Condition 17.6(b), in relation to the allocation of reviewing examiners to review of marking cases. The action plan is split into two linked sections:

  • the high level action plan for achieving compliance by the next examination series
  • a longer term action plan for series following the next examination series

Proposed short term solution

The following section outlines our action plan for securing compliance with GCSE/GCE Condition 17.6(b) by the time of the next post results period following the autumn examination series.

The action plan provides details of the solution that is proposed and the IT development and changes in our way of working that will be required to achieve it. Timelines are given later on in this document.

Summary of the short term solution

There are two strands to the work to achieve compliance for the next examination series:

  • availability of examiner resource to complete reviews of marking, and increasing the likelihood that sufficient compliant reviewers can be recruited from the pool of examiners used in the summer, and;

  • updates to systems in order to ensure that any potential non-compliant allocation of reviews of marking are not allocated, and creating an exception system whereby scripts which cannot be allocated to an examiner who has not marked any part of it before are allocated to two, or more, different examiners who then review the parts of it that they have not marked before.

Making better use of the examining workforce
Alteration of the process for reviewing the auto generated forecast

Prior to each examination series, a forecast of examiner demand is created automatically. The forecast is based on an analysis of the number of expected entries for each component, the number of discrete items within the assessment for each component, the type of examiner that is needed to mark the items within the assessment, and a reasonable workload for each examiner in the period available for marking. In this way, an estimate is generated of the number of examiners required and the number of items each of the examiners will mark.

When the forecast is generated, subject teams are required to review it to ensure that there are enough examiners to complete the work, that examiner workload is reasonable, and that the work can be completed in the time available. The forecast is updated, if necessary, following this review.

Subject teams will now be required, in addition to the above, to analyse the forecast to ensure there is a sufficient pool of review examiners which are available during the Post Results Services window because of the condition that reviewers must not have had involvement with the original assessment of the candidate work, even when they had only marked a proportion of the candidate work originally. This work had already been completed for the Sumner 2020 examination series prior to its cancellation. A similar process will be followed for the next GQ examination series.

Criteria for reviewing the forecast through the PRS lens

Subject teams will be briefed, ahead of the next available examination series, to review the forecast of examiner demand to ensure that as far as possible the pool of reviewers is sufficient to ensure that item marked scripts can be reviewed in a compliant manner.

Subject teams will give particular attention to the following areas when reviewing and updating the forecast. Additional information, such as the volume of items which are forecast to be marked by senior AA s, and the volumes of PRS requests in previous years, will also be considered.

Chief and PE roles are covered by the same person - The Chief Examiner role is usually filled from the ranks of Principal Examiners. As such, for at least one component in each qualification, the Chief and Principal Examiner will be the same person. In such cases, the PRS forecast will be amended so that the PE from another paper can review PRS cases.

Sole marked - the PE will not be eligible to conduct PRS cases; teams will recruit the Chief or Chairs of Examiners to complete ROMs and/or recruit another examiner and update the re- forecast. Consideration will have to be given to how additional examiners or other senior examiners are standardised when no standardisation meeting has been scheduled.

Smaller entry, but not sole marked - there is a chance that for some ROM cases, neither the PE or the examiner(s) on the paper are eligible to conduct them as between them all, they marked some of the original script. Consideration will be given to recruiting additional AAs to mitigate this risk and/or reduce the allocation of the PE so that the chance of them being able to complete ROM cases increases. When taking this action, teams must balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding.

Small entry where the PE/TL s would usually mark the graduate items - the PE, in this case, will not be able to mark any ROM cases because they marked all of the graduate items. Recruit a graduate marker or increase the number of examiners so that they can share the graduate items between them.

Larger entry papers - there is a decreased risk that ROM cases are not able to be allocated to the TLs, PAs and PEs and above, but the risk remains, particularly if there are large scale mark reviews such as borderlining. Reduce the allocation size of the senior examiners in order to increase the chance that they are able to conduct the majority of ROMs and also ensure there are sufficient graduate markers contracted to cover the total graduate allocation. When thinking about a reduced allocation size, balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding. When considering whether to conduct borderlining, decide whether the benefit of borderlining is greater than the risk to delivery of ROM cases.

Recruitment of additional reviewers

In some cases, it will be necessary, in addition to the steps outlined above, to increase the size of the examining pool so that sufficient experienced reviews are available for the purposes of PRS reviews of marking.

The forecast demand will be adjusted and additional reviews recruited through business as usual processes. In shortage areas, such as lesser taught languages in Modern Foreign Languages, where there is a scarcity of personnel with sufficient skills and experience, a combination of the measures outlined in this action plan will have to be enacted.

Alteration to the PRS forecast

As per normal business processes, when an examination series is nearing completion, a further forecast is generated which automatically estimates the demand for the reviewers needed to complete review of marking cases. The forecast, at present, is based on an analysis of the percentage of entry of the volume of reviews of marking from the previous equivalent exam, which is applied to the present series to generate the estimated demand for reviewers.

The PRS forecast represents another opportunity in which to ensure that the pool of reviewers is sufficient to ensure that reviews are conducted compliantly. The forecast data will therefore be amended to display the total proportion of items/scripts which the current pool of senior examiners and other nominated reviewers had marked originally so that, where needed, the pool can be added to.

Contingency exception process

There will remain a need to ensure there is a contingency arrangement in place should the actions taken above fail to completely mitigate the risk of there being insufficient examiners available in order to complete reviews of marking in a compliant and timely manner. As such, an exception process is being developed and implemented whereby reviews of marking can be reviewed by item. This exception process, and the adaptations to technology that are required for it, is detailed below.

Recruitment: Summary of actions to be taken against the conditions to ensure compliance

The table below summarises the actions that will be taken to improve recruitment practices to ensure that a sufficient pool of reviews will be available during the post results window following the autumn series.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (a) all reviews of marking of Marked Assessment Material will be carried out by Assessors who have appropriate competence and who have no personal interest in the outcome of the review being carried out,
Pearson actions Progress to date
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the exam series AA demand forecast to ensure that sufficient AAs are recruited for ROMs This was completed for the cancelled summer 2020 exam series. This will now be planned for to be repeated in November of 2020 for the autumn exam series
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the PRS forecast so that in the case of small or sole marked papers, additional reviewers are recruited, or drawn from the same team of senior examiners This will be put in plan for November so that staff are trained to review the forecast and recruit additional AAs
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by recruiting sufficient examiners for larger papers Teams will be trained to recruit sufficient AAs to complete reviews of marking ahead of the autumn series
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by ensuring that examiners have no personal interest in the review through current recruitment practices ROM reviewers will be recruited through the current process which ensures they have no personal interest
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by conducting compulsory training and standardisation for reviewers prior to completing ROMs Reviewers will be trained and standardised in November. Reviewers will not be able to conduct ROMs until standardisation has been satisfactorily been completed
Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (a) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task
Pearson actions Progress to date
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by recruiting additional examiners so that the pool is sufficient to ensure assessors do not conduct ROMs on all or parts of scripts they had assessed before As above, forecasts will be reviewed and additional AAs recruited
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring there is management oversight of reviewer recruitment plans and forecast Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring that ROM allocations are not made to original assessors The actions in relation to this are detailed more thoroughly below. Reviewers will be trained so that they understand that they must only complete ROMs on scripts, or parts of scripts, they have not assessed before. Reviewers will be further trained on the processes and systems that ensure they cannot do this.
Allocating reviews when all examiners have marked some part of the original script

We expect that the majority of review of marking (ROM) cases will be allocated as normal. For the cases that cannot be allocated as normal as there are no review markers available who have not marked some or all of the script previously, an exception process will be deployed. The exception process, which is under development, will allow the review to be conducted by 2 or more reviewers in order that each of the review examiners is only making a judgement on the parts of the script they haven’t seen.

The purpose of handling the script differently is to allow the script to be first reviewed by an examiner who has marked the least number of items originally, and once the examiner has submitted marks for the ROM allocation, for our ROM systems to keep the case in quarantine, with the aim of getting the non-compliant items (items originally marked by the reviewing examiner) reviewed by another examiner. The outcome of the review of marking would then be reviewed, if necessary, and published to the centre after the review on the non-compliant items has been completed by another examiner.

Handling these exception scripts differently will involve making some significant changes to our ROM systems to allow for 2 allocation scenarios:

Scenario 1 - Business as usual. The ROM system attempts to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation, and finds a suitable examiner. The review is completed as normal.

Scenario 2 - Exception process. The ROM system fails to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation. The following steps will be executed.

  • the request is not auto allocated and need to be managed as an exception script for manual examiner allocation
  • processing teams will manually identify a suitable examiner to complete the review based on which of them has marked the least amount of items originally, taking into account the seniority level of the original examiner
  • processing teams will manually allocate the script to the identified examiner using the ROM system
  • the system will display a warning to the operator that they are assigning a non-compliant allocation
  • the request is allocated to a review examiner
  • the system will record that the allocation is a non-compliant allocation
  • the system will flag the allocation so that it knows to handle the script differently
  • The first review examiner acknowledges the allocation, completes the review and submits a new paper mark
  • the new mark submitted by the review examiner will not be published and the mark will be kept in a quarantine area for another review to be completed on the items which the first reviewer had marked originally.
  • once the additional review on the item(s) which the first reviewer had marked originally has been completed, assessment teams will review and publish the ROM mark from a quarantine area which will update the paper mark for the review cases directly
  • the new mark is graded and the outcome of the review will be issued for the request if all criteria to generate an outcome has been met

In this way, as described above, the items which the first reviewer had marked originally will be reviewed by a second reviewer who had no involvement in the original assessment of the items. The scores given by the second reviewer for these items will be aggregated with those of the first so that each item on the review script is reviewed by an examiner who had not seen the items before.

Timelines for short term solution - Workaround for Item Marking
Reviewing examiner availability and recruitment
Status Task Expected completion date in 2020*
Complete Review and adjustment of Summer 2020 examination series forecast Closed
Not due Review and adjustment of the forecast for the next examination series Sept 10
Not due Recruitment of additional examiners for pinch points Sept 10 - Oct 05
Not due Review of the PRS forecast for the next examination series Nov 21
Not due Further recruitment activity for ‘pinch points’ Nov 21 - Dec 16

*At the time of writing the operational calendar for the autumn series was still being reviewed. The dates given here are therefore provisional and subject to change.

Systems and Process updates
System Timeframe for scoping, development, testing and implementation Expected Completion date in 2020*
iSeries Work to take a total of 13 weeks to complete. Delivery of the updates to the iSeries will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems. Nov 30
EOL Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems. Nov 30
Online Marking Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems. Nov 30

*At the time of writing the teams involved in this work are also heavily involved in working on different solutions for summer grading and the autumn series due to Covid-19. The dates given here are therefore provisional and subject to change.

Conducting reviews of marking: summary of actions to be taken against the conditions to ensure compliance

In addition to the recruitment actions to ensure compliance that are identified, above, the following actions will be taken to ensure that reviews of marking are carried out in accordance with the conditions.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

(b) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task,

Pearson actions Progress to date
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that auto allocations to non- compliant reviewers are blocked This work has been scoped and is in plan to commence in the next three weeks, with full delivery prior to the autumn series, by no later than Nov 30
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by providing training to our processing staff about the action taken when scripts need to be allocation by exception to more than one reviewer Processing teams have been fully briefed on the requirements of the conditions and will be trained prior to the next examination series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that manual allocation to more than one reviewer is possible by exception This work has been scoped and is in plan to commence in the next three weeks, with full delivery prior to the autumn series, by no later than Nov 30
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training our assessors to understand the exception ROM process and the systems involved with it Prior to each series reviewing assessors are trained and must pass standardisation. The training will be adapted this autumn so that it makes clear to them the requirements of the conditions and the processes that must be followed.
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training assessment staff about the new process and the requirement for them to review and approve candidate outcomes Assessment teams have been fully briefed on the requirements of the conditions and will be trained prior to the next examination series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by adapting systems so that non-compliant reviews that are made by mistake are recorded and escalated for remedial action This work has been scoped and is in plan to commence in the next three weeks, with full delivery prior to the autumn series, by no later than Nov 30
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the allocation of reviews to ensure compliance and to escalate allocations which are not compliant Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the completion of reviews which are conducted by item to ensure that assessors review the parts of scripts they have not assessed before Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window

Long term solution

Summary of long term solution

The process described above to manage reviews of marking which cannot be allocated to a compliant reviewer will be labour and time intensive. As such we are scoping the development needed to the pre existing functionality within our ePen marking platform with a view to enabling and testing it for use in a future live series.

The pre existing system allows assessments which were previously marked onscreen to be reviewed by item. Following centre requests for reviews of marking though our post results services, each script will be broken down into its constituent parts and fed into the system.

Reviewing examiners who have been successfully standardised will access the item reviewing system and will be able to view the pool of items that are available to be reviewed for any component they have been contracted for. Reviewers will only be able to view and mark items for which they had not previously had any involvement in assessing.

When the review of each item is completed, the marks will be aggregated and the outcome communicated to the requesting centre.

This system is currently configured so that reviewing examiners can only award a score which is equal to or greater than the original score. Development work will be required to update the system so that all score points are available to the reviewing examiner. Furthermore, other systems will need to be adapted so that they are compatible.

Timelines

At present, we provisionally plan full implementation of item reviewing for summer 2021, though this will be kept under review to ensure systems and processes can be developed and tested in time to minimise the risk of a major change to the manner in which we operate reviews of marking. It is not possible, given the rapidly changing nature of the autumn examination series, to commit to an earlier series.

The solution we present above for the manual allocation of review cases in the next examination series will be kept live as a contingency for future series when the item review system is launched.

The item-review system, the adaptations to it and other systems, and the changes to processes are still in the process of being scoped and development work scheduled. At present, therefore, we do not have firm estimates for the timelines for development and implementation.

Long term systems updates: summary of actions to be taken against the conditions to ensure compliance

In addition to the actions described above in relation to the recruitment and completion of ROMs, the following systems updates will be made to ensure ongoing compliance with GCSE/GCE condition 17.6 in future series

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (a) all reviews of marking of Marked Assessment Material will be carried out by Assessors who have appropriate competence and who have no personal interest in the outcome of the review being carried out,
Pearson actions Progress to date
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by updating our processes and systems for generating forecasts Current forecasts for in-series item marking will be adapted so that item reviewing forecasts can be generated
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by briefing staff on the forecast and systems for item reviewing All teams will be trained on the requirements of the item forecast for ROMs on the item review system
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by recruiting sufficient review assessors to conduct ROMs by item Current recruitment practices as for an exam series will be adapted for recruitment of reviewers for item reviewing
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by briefing the reviewing assessors on the requirements for item reviewing prior to the series in which it is implemented All reviewers must complete mandatory standardisation and training prior to completing ROMs. The training will be adapted to include training on the usage of the new system.
Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (a) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task,
Pearson actions Progress to date
Will will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by implementing item reviewing so that the original assessor will not be able to view or review items they had marked originally Systems and processes are being scoped for implementation in a future exam series

Annex C: Updated Action Plan (August 2021)

Pearson Action Plan for PRS compliance

Action plan

We outline here our high level action plan, with associated estimated timelines, to achieve compliance with the condition, GCSE/GCE Condition 17.6(b), in relation to the allocation of reviewing examiners to review of marking cases. The action plan is split into two linked sections:

  • the high level action plan for achieving compliance by the next examination series
  • a longer term action plan for series following the next examination series
Proposed short term solution

The following section outlines our action plan for securing compliance with GCSE/GCE Condition 17.6(b) by the time of the next post results period following the autumn examination series.

The action plan provides details of the solution that is proposed and the IT development and changes in our way of working that will be required to achieve it. Timelines are given later on in this document.

Summary of the short term solution

There are two strands to the work to achieve compliance for the next examination series:

  1. (a) availability of examiner resource to complete reviews of marking, and increasing the likelihood that sufficient compliant reviewers can be recruited from the pool of examiners used in the summer, and;

  2. (b) updates to systems in order to ensure that any potential non-compliant allocation of reviews of marking are not allocated, and creating an exception system whereby scripts which cannot be allocated to an examiner who has not marked any part of it before are allocated to two, or more, different examiners who then review the parts of it that they have not marked before.

Making better use of the examining workforce
Alteration of the process for reviewing the auto generated forecast

Prior to each examination series, a forecast of examiner demand is created automatically. The forecast is based on an analysis of the number of expected entries for each component, the number of discrete items within the assessment for each component, the type of examiner that is needed to mark the items within the assessment, and a reasonable workload for each examinerin the period available for marking. In this way, an estimate is generated of the number of examiners required and the number of items each of the examiners will mark.

When the forecast is generated, subject teams are required to review it to ensure that there are enough examiners to complete the work, that examiner workload is reasonable, and that the work can be completed in the time available. The forecast is updated, if necessary, following this review.

Subject teams will now be required, in addition to the above, to analyse the forecast to ensure there is a sufficient pool of review examiners which are available during the Post Results Services window because of the condition that reviewers must not have had involvement with the original assessment of the candidate work, even when they had only marked a proportion of the candidate work originally. This work had already been completed for the Sumner 2020 examination series prior to its cancellation. A similar process will be followed for the next GQ examination series.

Criteria for reviewing the forecast through the PRS lens

Subject teams will be briefed, ahead of the next available examination series, to review the forecast of examiner demand to ensure that as far as possible the pool of reviewers is sufficient to ensure that item marked scripts can be reviewed in a compliant manner.

Subject teams will give particular attention to the following areas when reviewing and updating the forecast. Additional information, such as the volume of items which are forecast to be marked by senior AAs, and the volumes of PRS requests in previous years, will also be considered.

Chief and PE roles are covered by the same person - The Chief Examiner role is usually filled from the ranks of Principal Examiners. As such, for at least one component in each qualification, the Chief and Principal Examiner will be the same person. In such cases, the PRS forecast will be amended so that the PE from another paper can review PRS cases.

Sole marked - the PE will not be eligible to conduct PRS cases; teams will recruit the Chief or Chairs of Examiners to complete ROMs and/or recruit another examiner and update the re- forecast. Consideration will have to be given to how additional examiners or other senior examiners are standardised when no standardisation meeting has been scheduled.

Smaller entry, but not sole marked - there is a chance that for some ROM cases, neither the PE or the examiner(s) on the paper are eligible to conduct them as between them all, they marked some of the original script. Consideration will be given to recruiting additional AAs to mitigate this risk and/or reduce the allocation of the PE so that the chance of them being able to complete ROM cases increases. When taking this action, teams must balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding.

Small entry where the PE/TLs would usually mark the graduate items - the PE, in this case, will not be able to mark any ROM cases because they marked all of the graduate items. Recruit a graduate marker or increase the number of examiners so that they can share the graduate items between them.

Larger entry papers - there is a decreased risk that ROM cases are not able to be allocated to the TLs, PAs and PEs and above, but the risk remains, particularly if there are large scale mark reviews such as borderlining. Reduce the allocation size of the senior examiners in order to increase the chance that they are able to conduct the majority of ROMs and also ensure there are sufficient graduate markers contracted to cover the total graduate allocation. When thinking about a reduced allocation size, balance the risk to PRS with the need for PEs, for example, to complete enough first line marking that they can meaningfully participate in awarding. When considering whether to conduct borderlining, decide whether the benefit of borderlining is greater than the risk to delivery of ROM cases.

Recruitment of additional reviewers

In some cases, it will be necessary, in addition to the steps outlined above, to increase the size of the examining pool so that sufficient experienced reviews are available for the purposes of PRS reviews of marking.

The forecast demand will be adjusted and additional reviews recruited through business as usual processes. In shortage areas, such as lesser taught languages in Modern Foreign Languages, where there is a scarcity of personnel with sufficient skills and experience, a combination of the measures outlined in this action plan will have to be enacted.

Alteration to the PRS forecast

As per normal business processes, when an examination series is nearing completion, a further forecast is generated which automatically estimates the demand for the reviewers needed to complete review of marking cases. The forecast, at present, is based on an analysis of the percentage of entry of the volume of reviews of marking from the previous equivalent exam, which is applied to the present series to generate the estimated demand for reviewers.

The PRS forecast represents another opportunity in which to ensure that the pool of reviewers is sufficient to ensure that reviews are conducted compliantly. The forecast data will therefore be amended to display the total proportion of items/scripts which the current pool of senior examiners and other nominated reviewers had marked originally so that, where needed, the pool can be added to.

Contingency exception process

There will remain a need to ensure there is a contingency arrangement in place should the actions taken above fail to completely mitigate the risk of there being insufficient examiners available in order to complete reviews of marking in a compliant and timely manner. As such, an exception

process is being developed and implemented whereby reviews of marking can be reviewed by item. This exception process, and the adaptations to technology that are required for it, is detailed below.

Recruitment: Summary of actions to be taken against the conditions to ensure compliance

The table below summarises the actions that will be taken to improve recruitment practices to ensure that a sufficient pool of reviews will be available during the post results window following the autumn series.

Condition of recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (a) all reviews of marking of Marked Assessment Material will be carried out by Assessors who have appropriate competence and who have no personal interest in the outcome of the review being carried out,
Pearson actions Progress to date
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the exam series AA demand forecast to ensure that sufficient AAs are recruited for ROMs This was implemented for the Autumn 2020 series and will be put in place again for the Autumn 2021 series.
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by training staff to review the PRS forecast so that in the case of small or sole marked papers, additional reviewers are recruited, or drawn from the same team of senior examiners This was implemented for the Autumn 2020 series and will be put in place again for the Autumn 2021 series.
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by recruiting sufficient examiners for larger papers A sufficient number of examiners were recruited for larger papers as required y the additional PRS forecast
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by ensuring that examiners have no personal interest in the review through current recruitment practices ROM reviewers were recruited through standard practices to ensure no examiners allocated a review had a personal interest in the review
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by conducting compulsory training and standardisation for reviewers prior to completing ROMs  
Condition of recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (b)an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved ina review of marking of the Learner’s Marked Assessment Material in respect of that task,
Pearson actions Progress to date
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by recruiting additional examiners so that the pool is sufficient to ensure assessors do not conduct ROMs on all or parts of scripts they had assessed before As above, forecasts were reviewed and additional AAs recruited as required
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring there is management oversight of reviewer recruitment plans and forecast Additional reporting will be established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are conducted by assessors who were not involved in the original marking by ensuring that ROM allocations are not made to original assessors The actions in relation to this are detailed more thoroughly below. Reviewers were trained for the Autumn 2020 exam series so that they understand that they must only complete ROMs on scripts, or parts of scripts, they have not assessed before. Reviewers also received further training on the processes and systems that ensure they cannot do this.
Allocating reviews when all examiners have marked some part of the original script

We expect that the majority of review of marking (ROM) cases will be allocated as normal. For the cases that cannot be allocated as normal as there are no review markers available who have not marked some or all of the script previously, an exception process will be deployed. The exception process, which is under development, will allow the review to be conducted by two or more reviewers in order that each of the review examiners is only making a judgement on the parts of the script they haven’t seen.

The purpose of handling the script differently is to allow the script to be first reviewed by an

examiner who has marked the least number of items originally, and once the examiner has submitted marks for the ROM allocation, for our ROM systems to keep the case in quarantine, with the aim of getting the non-compliant items (items originally marked by the reviewing examiner) reviewed by another examiner. The outcome of the review of marking would then be reviewed, if necessary, and published to the centre after the review on the non-compliant items has been completed by another examiner.

Handling these exception scripts differently will involve making some significant changes to our ROM systems to allow for two allocation scenarios.

Scenario 1 - Business as usual. The ROM system attempts to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation, and finds a suitable examiner. The review is completed as normal.

Scenario 2 - Exception process. The ROM system fails to find a suitable examiner who has not been assigned as a marker for all items on the script at the time of allocation. The following steps will be executed.

  • the request is not auto allocated and need to be managed as an exception script for manual examiner allocation

  • processing teams will manually identify a suitable examiner to complete the review based on which of them has marked the least amount of items originally, taking into account the seniority level of the original examiner
  • processing teams will manually allocate the script to the identified examiner using the ROM system
  • the system will display a warning to the operator that they are assigning a non-compliant allocation
  • the request is allocated to a review examiner

  • the system will record that the allocation is a non-compliant allocation

  • the system will flag the allocation so that it knows to handle the script differently
  • The first review examiner acknowledges the allocation, completes the review and submits a new paper mark
  • the new mark submitted by the review examiner will not be published and the mark will be kept in a quarantine area for another review to be completed on the items which the first reviewer had marked originally.
  • once the additional review on the item(s) which the first reviewer had marked originally has been completed, assessment teams will review and publish the ROM mark from a quarantine area which will update the paper mark for the review cases directly
  • the new mark is graded and the outcome of the review will be issued for the request if all criteria to generate an outcome has been met

In this way, as described above, the items which the first reviewer had marked originally will be reviewed by a second reviewer who had no involvement in the original assessment of the items. The scores given by the second reviewer for these items will be aggregated with those of the first so that each item on the review script is reviewed by an examiner who had not seen the items.

Timelines for short term solution - Workaround for Item Marking

Reviewing examiner availability and recruitment
Status Task Expected completion date in 2020*
Complete Reviewing examiner availability and recruitment Closed  
Complete Review and adjustment of the forecast for the next examination series Closed  
Complete Recruitment of additional examiners for pinch points Closed  
Complete Review of the PRS forecast for the next examination series Closed  
Complete Further recruitment activity for ‘pinch points’ Closed  
Systems and Process updates
System Timeframe for scoping, development, testing and implementation Expected Completion date in 2020*
iSeries Work to take a total of 13 weeks to complete. Delivery of the updates to the iSeries will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems. Closed
EOL Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems. Closed
Online Marking Work to take a total of 24 ‘person weeks’ (total of 120 hours). Delivery of the updates to the EOL and Online Marking will be aligned to complete at the same time as the other systems, because of the interdependencies across the systems. Closed

Conducting reviews of marking: summary of actions to be taken against the conditions to ensure compliance

In addition to the recruitment actions to ensure compliance that are identified on page 6, above, the following actions will be taken to ensure that reviews of marking are carried out in accordance with the conditions.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

b.) an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task,

Pearson actions Progress to date
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that auto allocations to non- compliant reviewers are blocked This work was completed in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by providing training to our processing staff about the action taken when scripts need to be allocation by exception to more than one reviewer Training was provided to all processing staff in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by updating our systems so that manual allocation to more than one reviewer is possible by exception This work was completed in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training our assessors to understand the exception ROM process and the systems involved with it Training was provided to all assessors in advance of the Autumn 2020 series & only those that passed standardisation were allocated work. The training was adapted so that it makes clear tothem the requirements of the conditions and the processes that must be followed.
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by training assessment staff about the new process and the requirement for them to review and approve candidate outcomes Assessment teams have been fully briefed on the requirements of the conditions trained prior to the next examination series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by adapting systems so that non-compliant reviews that are made by mistake are recorded and escalated for remedial action This work was completed in advance of the Autumn 2020 series
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the allocation of reviews to ensure compliance and to escalate allocations which are not compliant Additional reporting was established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by monitoring the completion of reviews which are conducted by item to ensure that assessors review the parts of scripts they have not assessed before Additional reporting was established for the autumn series onward so that resource plans are reviewed at regular Closure and Completion meetings prior to and during the post results services window

Long term solution

Summary of long term solution

The process described above to manage reviews of marking which cannot be allocated to a compliant reviewer will be labour and time intensive. As such we are scoping the development needed to the pre existing functionality within our ePen marking platform with a view to enabling and testing it for use in a future live series.

The pre existing system allows assessments which were previously marked onscreen to be reviewed by item. Following centre requests for reviews of marking though our post results services, each script will be broken down into its constituent parts and fed into the system.

Reviewing examiners who have been successfully standardised will access the item reviewing system and will be able to view the pool of items that are available to be reviewed for any component they have been contracted for. Reviewers will only be able to view and mark items for which they had not previously had any involvement in assessing.

When the review of each item is completed, the marks will be aggregated and the outcome communicated to the requesting centre.

This system is currently configured so that reviewing examiners can only award a score which is equal to or greater than the original score. Development work will be required to update the system so that all score points are available to the reviewing examiner. Furthermore, other systems will need to be adapted so that they are compatible.

Timelines

At present, we provisionally plan full implementation of item reviewing for summer 2021, though this will be kept under review to ensure systems and processes can be developed and tested in time to minimise the risk of a major change to the manner in which we operate reviews of marking. It is not possible, given the rapidly changing nature of the autumn examination series, to commit to an earlier series.

The solution we present above for the manual allocation of review cases in the next examination series will be kept live as a contingency for future series when the item review system is launched.

The item-review system, the adaptations to it and other systems, and the changes to processes are still in the process of being scoped and development work scheduled. At present, therefore, we do not have firm estimates for the timelines for development and implementation.

Long term systems updates: summary of actions to be taken against the conditions to ensure compliance

In addition to the actions described above in relation to the recruitment and completion of ROMs, the following systems updates will be made to ensure ongoing compliance with GCSE/GCE condition 17.6 in future series.

Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (a) all reviews of marking of Marked Assessment Material will be carried out by Assessors who have appropriate competence and who have no personal
Pearson actions Progress to date
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by updating our processes and systems for generating forecasts Current forecasts for in-series item marking will be adapted so that item reviewing forecasts can be generated
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by briefing staff on the forecast and systems for item reviewing All teams will be trained on the requirements of the item forecast for ROMs on the item review system
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by recruiting sufficient review assessors to conduct ROMs by item Current recruitment practices as for an exam series will be adapted for recruitment of reviewers for item reviewing
We will ensure that sufficient reviewers with appropriate competence and no personal interest in the outcome of the review being carried out are recruited by briefing the reviewing assessors on the requirements for item reviewing prior to the series in which it is implemented All reviewers must complete mandatory standardisation and training prior to completing ROMs. The training will be adapted to include training on the usage of the new system.
Condition of Recognition

GCSE (9‐ 1) 17.6; and GCE (reform) 17.6 The arrangements must provide that –

  1. (a) - an Assessor who was previously involved in the marking of a task in an assessment in respect of a Learner must not be involved in a review of marking of the Learner’s Marked Assessment Material in respect of that task,
Pearson actions Progress to date
We will ensure that reviews of marking are allocated to reviewers that had no prior involvement in the marking of the task by implementing item reviewing so that the original assessor will not be able to view or review items they had marked originally Systems and processes are being scoped for implementation in a future exam series

  1. As in force in Summer 2019. See Annex A, paragraphs 17-23 for further details. ↩

  2. Annex A – Legal Provisions ↩

  3. Annex B – Undertaking and Action Plan (June 2020) ↩

  4. As required by GCE / GCSE Qualification Level Conditions and Requirements 17.6(b) ↩

  5. As required by GCE / GCSE Qualification Level Conditions and Requirements 17.6(a) ↩

  6. As required by General Condition A5.2(a) ↩

  7. As required by General Condition A5.2(e) ↩

  8. As required by General Condition A6.1 ↩

  9. The payment of compensation to Centres will take place after the publication of this Notice [on a date to be determined]. ↩

  10. These reasons are set out in more detail in the ‘Relevant Cases’ section of this Notice. ↩

  11. Page 36 ↩

  12. Page 63 ↩

  13. Table, page 63 ↩

  14. Data provided by Pearson 6 March 2020. ↩

  15. The item level review figures (for 2019 only) include a subset of affected item marked cases in which the person who conducted a quality assurance check before the issue of results (as opposed to the first line marking) was the same person who completed the review of marking. ↩

  16. The item level review figures (for 2019 only) include a subset of affected item marked cases in which the person who conducted a quality assurance check before the issue of results (as opposed to the first line marking) was the same person who completed the review of marking. ↩

  17. Annex B – Undertaking and Action Plan (June 2020) ↩

  18. Annex C – Updated Action Plan (August 2021) ↩

  19. Or as amended from time to time. ↩

  20. These are Qualification Level Conditions and Requirements that were in force in 2019 for GCE and GCSE (9-1) qualifications. There were a variety of Conditions in force during 2016, 2017 and 2018, depending on whether the affected qualifications were legacy or reformed qualifications. For the purposes of this undertaking, there is no material difference between the relevant Conditions. ↩

  21. Pearson Action Plan for PRS compliance, 28 April 2020 ↩

Back to top