Consultation outcome

Consultation on updates to the compliance and review methodology

Updated 31 January 2020

Compliance and review methodology

1. Background

1.1 The SSRO is proposing to update its compliance and review methodology (‘the methodology’). We are consulting publicly on a revised version of the methodology and we welcome responses to the specific questions set out below, as well as any general comments on the methodology.

1.2 We issued a working paper to key stakeholders in relation to this topic on 1 July 2019. We received written responses to the working paper from five organisations, representing the views of three contractors, the MOD and a trade organisation. The working paper was discussed at a workshop held at the SSRO’s offices on 10 July 2019, attended by representatives from the MOD, trade and contractor organisations.

1.3 We would like to thank respondents for sharing their views on the working paper. We have considered all submissions in drafting this consultation document and we have provided a summary of the main points raised in response to the working paper, and how we have sought to address these.

1.4 This consultation document was issued on 14 October 2019. Any representations should be sent to [email protected] by Friday 6 December 2019, using the consultation response form as set out below. Members of the SSRO’s Reporting and IT Sub-Group of the Operational Working Group will have the opportunity to discuss this consultation at the SSRO’s offices on 5 November 2019.

1.5 Any comments will inform our consideration of the need for changes to the updated methodology as drafted. The SSRO aims to allow reasonable time between publishing any changes and when they come into effect. The current timetable for publication of the revised methodology is 31 January 2020, with implementation from 1 April 2020. However, as part of this consultation, we invite representations on the timing of publication and implementation.

2. Key changes to the methodology

2.1 The SSRO has been operating its current compliance and review methodology since January 2017. In that time, it has gained a greater understanding of how the regulatory framework is being applied and has continued to develop its approach to implementing the methodology.

2.2 The methodology has been updated to acknowledge that compliance monitoring may inform the SSRO’s understanding of how the regulatory framework is being applied, as well as setting out the potential for compliance monitoring to impact more widely on the operation of the regime, specifically by:

• aiming to contribute to achieving good quality data from contractors;
• working to develop a shared understanding with stakeholders about identified issues and what appropriate action could be taken to address these issues; and
• informing changes to guidance issued by the SSRO, changes to the Defence Contracts Analysis and Reporting System or recommendations for legislative change.

2.3 Other updates include the reference to, and use of, the key developments that have taken place since January 2017. In particular, these relate to:

• a more risk-based approach with the consideration, primarily, of automatic validation warnings from the system that remain unresolved;
• the ability to raise issues within the system, and importantly to seek direct feedback from contractors or the MOD on issues arising;
• a continuing programme of engagement with the MOD on compliance, which has led to a greater reliance being placed on the verification of report submissions by the MOD against contract information, and which will inform our understanding of the application of the regime and of the quality of data within the system; and
• supplementing our routine work, which is based on the validation warnings in the system, by undertaking a deeper examination of how reporting requirements are being met in selected reports, either through ‘targeted’ or ‘thematic’ reviews, which will also provide feedback on how well the MOD reviews are working in practice.

2.4 Through these changes and the proposed updates to the methodology we plan to increasingly target important compliance issues whilst maintaining a clear focus on improvements that are required to data quality.

2.5 There remains a continued focus within the updated methodology on notifying the MOD of unresolved issues identified through the compliance monitoring process, where issues are not resolved or responded to by the contractor, or where the response appears inconsistent with the statutory guidance or legislation. It will only be through clear feedback loops that the SSRO will be able to understand whether reporting requirements have been met, or whether other aspects of the regulatory framework are being met or not.

2.6 The current methodology sets two key performance indicators, which inform the overall approach to compliance and the contents of the annual compliance report. The indicators measure the timeliness and quality of report submission and their use has provided an important basis for the development of the Annual Compliance Report. These indicators have provided an important basis for the development of the Annual Compliance Report, but it has proved difficult for contractors to meet reporting requirements first time with the result that the second indicator provides limited information about the extent of compliance. We have retained these, with minor amendments and added an additional indicator to provide insight into how contractors perform after the first report submission:

• 1a) “All required reports have been submitted within the relevant deadlines”;
• 2a) “Reporting obligations have been met first time for all reports submitted, in accordance with the Regulations and relevant statutory guidance”; and
• 2b) “Reporting obligations have been met either first time or in subsequent submissions for all reports submitted, in accordance with the Regulations and relevant statutory guidance”.

2.7 The methodology also notes that the SSRO’s approach to stakeholder engagement is important to support timely, good quality submissions, particularly as early engagement will assist the SSRO to identify the difficulties that contractors may have understanding, or complying with, reporting requirements.

3. Introduction

3.1 The Defence Reform Act 2014 (the Act) places two review obligations on the SSRO. Firstly, the SSRO is required, in accordance with section 36(2) of the Act, to keep under review the extent to which persons subject to reporting requirements are complying with them. Secondly, under section 39(1) of the Act, the SSRO is required to keep under review the provision of the regulatory framework established by Part 2 of the Act and the Single Source Contract Regulations 2014 (the Regulations). Reviewing and seeking to understand reports that are submitted in compliance with the legislation and the SSRO’s statutory guidance provides the SSRO with relevant information to discharge these functions.

3.2 We seek to use the methodology to demonstrate how the SSRO exercises its function under section 36(2), but also how our function to keep under review the provision of the regulatory framework established by the Act and the Regulations may be informed by information obtained from compliance monitoring.

3.3 The methodology emphasises that the SSRO does not have a general role to provide assurance that individual contracts have been priced in accordance with statutory requirements, nor does it seek to use its compliance monitoring function for this purpose.

3.4 Our working paper did not specifically ask for stakeholder views on the linkage between the SSRO’s legislative obligations and the objectives that we are seeking to achieve through this methodology. Although no general comments were raised in relation to this by respondents, we are interested in understanding stakeholder views on the exercise of these functions.

Question 1: Does the methodology clearly demonstrate how the SSRO will exercise its s36(2) function and how its s39(1) function may be informed though our work in this area?

4. The SSRO’s general approach

4.1 The SSRO proposes that the methodology should be updated to reflect actual practice and provide greater transparency regarding its approach. Some respondents to the working paper considered that the SSRO should detail the review process, while others felt that it should first be demonstrated that greater transparency as to how the SSRO logs issues and prioritises action will deliver benefits.

4.2 The SSRO is committed to transparency as one of its core values. We consider that working in partnership is more likely to lead to effective understanding of the extent to which reporting requirements are being met and result in data being reported that is used in support of the regulatory framework. Transparency will help all stakeholders to understand how the review process works and facilitate its delivery. We have drafted the methodology with distinct sections, detailing ‘the SSRO’s general approach’, ‘the review of report submissions’ and later ‘raising matters with the MOD’.

4.3 The first of these sets out the general approach that the SSRO will follow in reviewing submissions made by contractors, along with a commitment to rely on system automation as much as possible. It links the methodology to the vision set out in our data strategy, specifically that methodology aims to contribute to achieving good quality data from contractors that can be utilised in procurement decisions, contract management and in the development of the regulatory framework. It also sets out the three key performance indicators that the SSRO will apply when measuring compliance against statutory reporting requirements.

Question 2: Is the SSRO’s approach sufficiently clear from the methodology?

5. Review of report submissions

5.1 In response to the working paper two respondents stated that the SSRO should detail the review process, this section provides detail on the process that the SSRO will undertake when considering each submission. It notes that the SSRO will review all reports submitted and raise issues with contractors where appropriate. It also highlights the reliance we will place on the validation warnings within the system as well as the importance of the MOD’s assessment of a submission.

5.2 The exercise of our functions as detailed in the ‘Introduction’ section will only be properly achieved with significant input from stakeholders, particularly in response to the issues we raised on the system. These inputs are set out in the review process. It is our view that there should be a large degree of consistency in the reviews and in the categorisation of issues arising, irrespective of the submission being reviewed, as this will allow for a more complete record of issues to be analysed and considered. It is also our view that undertaking this work promptly will allow the opportunity for explanations to be provided early, informing potential actions to address issues.

Question 3: Is the SSRO’s review process sufficiently clear from the methodology?

5.3 The SSRO proposes that the methodology sets out the linkage between the compliance review process and the SSRO’s other functions. In particular it is suggested that the methodology makes clear that intelligence gathered from the compliance process will be used to inform future priorities for the SSRO, which should in turn improve the quality of the data held in the system.

5.4 Respondents to the working paper stated that it should first be demonstrated that greater transparency as to how the SSRO logs issues and prioritises action will deliver benefits. It is important to make clear that there may be ongoing work that arises from the findings of compliance reviews and that this will be appropriately prioritised.

5.5 Three respondents commented that the compliance process needs to be sufficiently flexible to deal with simple issues such as arithmetical or typographical errors, which may be resolved directly by a contractor, as well as to deal with more complex issues needing to be resolved in a different manner. The SSRO agrees with these representations and believes that the methodology appropriately acknowledges there may be a range of work needed by the SSRO to address complex issues.

Question 4: Do you agree that the methodology appropriately identifies how the findings from compliance reviews will inform the SSRO’s other work?

6. Additional compliance monitoring activities

6.1 The methodology sets out how the SSRO may seek to supplement its standard compliance reviews with additional targeted or thematic reviews.

6.2 One respondent to the working paper agreed that if an automated process was to be relied upon as part of the compliance approach then a risk-based approach to undertaking targeted reviews would be a means of validating whether the automated process was correct and provided the right emphasis. Another respondent noted that both targeted and thematic reviews could be useful.

6.3 Three respondents felt that these activities represented normal incremental improvements, evolution and quality assurance activities that would be expected with this type of process. They considered that the SSRO could proceed with the activities without the need to formalise them in the methodology. The SSRO accepts this is the case, but is committed to transparency and believes the methodology should reflect the various ways in which the SSRO discharges its functions. These activities are referred to in the methodology, without excessively detailing how they may be undertaken. This will allow for incremental developments to take place in future, without having to amend the methodology.

Question 5: Do you agree that these additional activities can be reflected in the methodology without setting out the detail of how each may be undertaken?

7. MOD reviews of report submissions

7.1 One respondent to the working paper stated that the SSRO should detail the links between the MOD’s review process and the compliance work undertaken by the SSRO, ensuring that they are clear and transparent. Three respondents stated that it would be inappropriate to link the SSRO’s work to the MOD’s reviews and thought that careful consideration would be required to ensure that the circumstances and the reasons for the MOD’s issues were understood fully before any further action could be taken.

7.2 The SSRO agrees that care needs to be taken to differentiate between its responsibilities and those of the MOD. There are, however, clear roles for both the SSRO and the MOD in relation to understanding the extent to which reporting requirements are being met and how the provision of the regulatory framework is being applied. For example, only the MOD is able to highlight any inconsistencies between what has been reported by the contractor and what has been contractually agreed to, particularly in relation to allowable costs and the contract profit rate.

7.3 The identification of any inaccuracies or inconsistencies with contractual information by the MOD, followed by subsequent corrections by contractors, will lead to improved data quality within the system. By undertaking analysis of the queries being raised by the MOD, and any subsequent amendments that are made by contractors, the SSRO will have a greater understanding of whether the information being provided is sufficient for the MOD’s contract management purposes. Where it is not, it will be able to consider whether it needs to undertake further work, such as an update to its guidance, to address such issues to prevent reoccurrence.

7.4 The methodology sets out an appropriate interface between the SSRO’s compliance reviews and the work of the MOD. It identifies that the SSRO will only undertake its reviews after giving the MOD opportunity to do so first. The SSRO will analyse issues raised by the MOD following its review of submissions, specifically to understand data quality issues within reports and also whether the information being provided is sufficient for the MOD’s contract management purposes, adding to the evidence base available to support the regulatory framework.

Question 6: Do you agree that the SSRO should only review a submission having given the MOD sufficient time to undertake its own review in the first instance?

8. Raising matters with the MOD

8.1 The methodology also sets out the circumstances in which the SSRO will notify the MOD of delayed submissions and of any specific issue that it has raised on a submission. This enables the MOD to provide feedback to inform the SSRO’s compliance reviews and to take any appropriate action.

8.2 Respondents to the working paper were generally of the view that the SSRO should not recommend compliance action to the MOD, as the SSRO would need to maintain an independent position should a matter be subsequently referred to it. The draft methodology does not provide for the SSRO to recommend that the MOD takes enforcement action, but ensures that the MOD has the relevant information to make an informed decision.

8.3 The methodology recognises that there may be a range of responses from the MOD to issues raised by the SSRO. It emphasises the importance of feedback from the MOD to ensure the SSRO understands the extent to which reporting requirements are being met and how the regulatory framework is being applied.

Question 7: What is your view on the general approach to the notification of issues to the MOD as reflected in the methodology?

9. Supporting compliance and improving the regime

9.1 With a view to understanding the reporting issues that a contractor may face and in turn improving the timeliness and data quality of submissions, the SSRO seeks to engage early in the reporting process with those subject to reporting obligations and the methodology sets out how we seek to undertake this.

9.2 Respondents to the working paper, and at the workshop, agreed that early engagement to support and enable compliance was a constructive development to the approach. They acknowledged positive previous experiences with the process for initial set up of a qualifying contract onto the system and with our engagement in the reporting of these contracts. Respondents also noted that early engagement by the SSRO and the capacity to enable contractors to gain early access to DefCARS has improved the quality of submissions. Three respondents to the working paper did, however, question whether it was necessary to formally reflect this focus in the methodology as this was good practice and would be carried out as a matter of course.

9.3 The SSRO does not have an express support function. We are, however, focused on exercising the statutory functions linked to our compliance methodology to encourage the use and quality of reported data as this lends itself to promoting our statutory aims. To this end, we may only provide support to the extent that it facilitates, or is conducive or ancillary to, one of our functions. It is the SSRO’s view that our:

• help desk assists people to understand our reporting guidance and also helps us understand how the regime is operating through the queries we receive;
• Reporting and DefCARS training and the use of automated validation warnings supports how we facilitate electronic submission of reports, compliance monitoring and analysis and that the training and analysis of non-resolved validation warnings provides opportunities for us to understand how the regime is operating; and
• on-boarding is in a similar category to the Reporting and DefCARS training.

9.4 Our view is that we should refer to this early engagement and support in the compliance methodology as it is helpful to link monitoring of reporting compliance with how we help people to make their submissions in the first instance.

Question 8: Are there any other elements of support or engagement that should be reflected in the methodology?

10. Reporting findings and providing feedback

10.1 The methodology sets out how the SSRO will provide feedback to the MOD and to contractors on issues identified.

10.2 The SSRO proposed some additional compliance metrics in the working paper and asked for stakeholder feedback on those and any other metrics that stakeholders would find useful. Respondents to the working paper, and at the workshop, questioned the value that additional metrics would add to the process and one respondent queried where ‘more strategic level’ reports would go. Respondents did, however, state that it would be useful to have a regular report to the contractor detailing themes arising in their report reviews and a summary of outstanding issues in order to allow them to gauge their own performance at an overall level. Respondents also highlighted that the ability to comment on outstanding issues and the ability to explain why they might still be outstanding would enrich the data available.

10.3 The methodology states that, in addition to the Annual Compliance Report, the SSRO will provide regular reports to the MOD, identifying where matters have had an impact either on data quality or the operation of the regulatory framework, and also to individual contractors with multiple contracts. One respondent noted that the provision of such reports will improve visibility of issues; understanding of error causes; the co-ordination of corrections and should ultimately lead to good quality data.

Question 9: What are the key areas of feedback for the SSRO to provide to industry?

11. Stakeholder input and next steps

11.1 A summary table of the questions for stakeholders is provided at Appendix 1 for ease of reference.

11.2 The SSRO welcomes views from interested parties on these and any other aspects of this consultation in writing by 6 December 2019 to [email protected]. Members of the Reporting and IT Sub-group will have an opportunity to discuss this consultation paper at our offices on 5 November 2019.

11.3 The SSRO also invites stakeholders to discuss any of the issues raised in this working paper with it on an individual basis. If you wish to do so, please contact us via [email protected].

Appendix 1: summary of questions.

Section heading	Question
Introduction	Question 1: Does the methodology clearly demonstrate how the SSRO will exercise its s36(2) function and how its s39(1) function may be informed though our work in this area?
The SSRO’s general approach	Question 2: Is the SSRO’s approach sufficiently clear from the methodology?
Review of report submissions	Question 3: Is the SSRO’s review process sufficiently clear from the methodology?
	Question 4: Do you agree that the methodology appropriately identifies how the findings from compliance reviews will inform the SSRO’s other work?
Additional compliance monitoring activities.	Question 5: Do you agree that these additional activities can be reflected in the methodology without setting out the detail of how each may be undertaken?
MOD reviews of report submissions	Question 6: Do you agree that the SSRO should only review a submission having given the MOD sufficient time to undertake its own review in the first instance?
Raising matters with the MOD	Question 7: What is your view on the general approach to the notification of issues to the MOD as reflected in the methodology?
Supporting compliance and improving the regime	Question 8: Are there any other elements of support or engagement that should be reflected in the methodology?
Reporting findings and providing feedback	Question 9: What are the key areas of feedback for the SSRO to provide to industry?