HMT: Correspondence Triage Automation Tool
Automates steps when triaging correspondence.
Tier 1 Information
Name
Correspondence Triage Automation Tool
Description
This algorithmic tool helps HM Treasury’s Correspondence and Information Rights team triage incoming Ministerial Correspondence, Treat Official & Private Office Actions. The tool predicts the following fields for each case: - Follow up - High risk - Meeting request - Responding minister - Standard lines - Summary - Subject - Team
The primary purpose of the triage automation tool is to ensure timely and accurate handling of correspondence by automating time-consuming tasks.
It may suggest standard lines where a routinely asked question is detected, but does not write the entire response, which is still prepared and signed off by a Treasury employee.
Website URL
N/A
Contact email
Tier 2 - Owner and Responsibility
1.1 - Organisation or department
HM Treasury
1.2 - Team
Correspondence and Information Rights Team, Data Science Hub
1.3 - Senior responsible owner
Chief Data Officer
1.4 - External supplier involvement
No
1.4.1 - External supplier
N/A
Tier 2 - Description and Rationale
2.1 - Detailed description
The triage automation tool makes use of three Artificial Intelligence (AI) and machine learning techniques.
Firstly, a Support Vector Machine algorithm is used for the classification of the correspondence team. Secondly, GPT-4 prompting is used for summarising the correspondence and for generating follow up, high risk, and meeting request flags.
Finally, Retrieval-Augmented Generation (RAG) is used to match the incoming correspondence with any appropriate standard lines.
2.2 - Scope
The triage automation tool has been designed to assist the Correspondence and Information Rights team in processing correspondence efficiently and effectively. This includes emails and scanned letters sent by email. The primary purpose of the triage automation tool is to ensure timely and accurate handling of correspondence by automating time consuming tasks.
2.3 - Benefit
The key benefits that the tool is expected to deliver include: improved efficiency in processing correspondence, saving several hours of manual work per day for multiple team members. Automated summarisation and flagging, which ensures timely identification of high-priority, high-risk, and meeting request correspondence, and the automatic matching of correspondence with appropriate standard responses. These together enable better prioritisation and management of correspondence which frees up team members to focus on more complex tasks.
2.4 - Previous process
The current process for triaging correspondence requires the same fields and categories to be identified by a member of the Correspondence and Information Team. This work was manually carried out by team members with automation limited to filling in basic details from the correspondence such as the sender’s name.
2.5 - Alternatives considered
We explored multiple methods before deciding upon the final multi-model approach. The final method was selected because it required minimal retraining and allowed for custom models for each of the subproblems.
Tier 2 - Decision making Process
3.1 - Process integration
Decisions regarding the processing of correspondence by the Correspondence and Information Rights team are made continuously, as new correspondence is received. The triage automation tool integrates into this decision-making process by automating the summarisation, allocation and standard line identification steps that were previously done manually, thereby enhancing efficiency and accuracy.
3.2 - Provided information
The tool predicts a text output for each of the following fields for each correspondence: - Follow up - High risk - Meeting request - Responding minister - Standard lines - Summary - Subject - Team
3.3 - Frequency and scale of usage
The triage automation tool is used automatically each time a Ministerial Correspondence, Treat Official & Private Office Action correspondence is received via the HMT email address.
3.4 - Human decisions and review
The predictions made by the triage automation tool are monitored by the member of the Correspondence and Information Rights team member who has been assigned the correspondence. They are then able to overwrite the predicted fields when needed. In addition to this, the performance of the triage automation tool is monitored by a project team. The accuracy of the tool is captured in an automatically updating report.
3.5 - Required training
Each Correspondence and Information Rights team member using the tool goes through an onboarding process that trains them on how to use the tool and troubleshoot.
3.6 - Appeals and review
N/A
Tier 2 - Tool Specification
4.1.1 - System architecture
The process begins with an API call triggered by a webhook, initiating data processing through several modules: - Summary: Generates a summary. - Flags: Identifies follow-up, high risk, and multi-issue flags. - Key Terms: Extracts key terms. - Meeting Request: Detects meeting requests. - Subject: Extracts the subject. - Standard Lines and Similarity Scores: Analyses for standard responses stored in a vector database in cloud storage and generates a cosine similarity score for each and then returns the most similar. - Top Teams and Probabilities: Determines top teams and probabilities. - Final Team Determination: Identifies the final responsible team. - Organisational Code Lookup: Uses an eCase API call to look up the organisational code. - Responding Minister Identification: Identifies the responding minister. - Output Generation: Compiles processed information and flags. - eCase API Call (Patch): Sends the final output back through an eCase API call (Patch) to update the system.
4.1.2 - Phase
Production
4.1.3 - Maintenance
The performance is monitored by a project team and collated automatically. This allows for maintenance to be carried out promptly when necessary. Data sources relied upon for standard lines predictions are updated dynamically as standard lines are added to the internal correspondence management software. In the future, the logic and rules employed by the tool may evolve over time as new secure LLM models become available.
4.1.4 - Models
GPT4 Turbo: a large language model developed by OpenAI, capable of understanding and generating human-like text based on the input it receives.
Support Vector Machine (SVM): a supervised learning algorithm used for classification and regression tasks, capable of finding the optimal hyperplane that separates different classes in the data.
Tier 2 - Model Specification
4.2.1 - Model name
Correspondence Triage Automation Tool
4.2.2 - Model version
v1.0
4.2.3 - Model task
To automatically classify, summarise, and flag correspondence received by the HMT email address, and match it with appropriate standard lines, aiding the Correspondence and Information Rights team in timely processing of correspondence.
4.2.4 - Model input
.msg or .eml file containing correspondence.
4.2.5 - Model output
json containing predictions for follow up flag, high risk flag, meeting request flag, responding minister, standard lines, summary, subject and team.
4.2.6 - Model architecture
Support vector machine is used for team. GPT4 is used for high risk flag, meeting request, summary and subject. A mapping from subject is used for responding minister. Standard Lines are predicted using a Retrieval-Augmented Generation process.
4.2.7 - Model performance
Using a randomised train-test split teams prediction (using only SVM without additional rule-based components) achieved ~70% accuracy. The components subject, responding minister, high-risk flag, meeting-request and summary have been directly evaluated as performing as good or better than existing processes. In addition to model performance, robustness tests have been carried out to assess the tool’s ability to integrate into the correspondence management software and access the API’s ability to remain functioning as intended during high traffic periods.
4.2.8 - Datasets
The model has been developed with two types of historic dataset: - Datasets containing previously received Ministerial Correspondence, Treat Official & Private Office Action correspondences classified by team. - Datasets containing standard lines and responses used in prior correspondences.
4.2.9 - Dataset purposes
The datasets containing previously received Ministerial Correspondence, Treat Official & Private Office Action correspondences classified by team where used to train, validate and test the performance of the model.
Tier 2 - Data Specification
4.3.1 - Source data name
Historical classified correspondence (eCase Bulk Report).
4.3.2 - Data modality
Tabular
4.3.3 - Data description
This data provides information about previous Ministerial Correspondence, Treat Official & Private Office Action correspondence sent to HMT including the contents of the correspondence and attachments and information about team, subject and responding minister.
4.3.4 - Data quantities
1,085,710 documents comprising .eml, .msg, .doc, .pdf and .img (252.07 GiB). After preprocessing, such as removing unreadable files, this dataset was used in testing with a 80/20 split in order to establish the Support Vector Machine model accuracy.
4.3.5 - Sensitive attributes
The sensitive attributes of these data sets are: - Name - Email Address - Correspondence content
4.3.6 - Data completeness and representativeness
The dataset is sampled to only include correspondence including .msg, .eml, .doc and .pdf files.
4.3.7 - Source data URL
N/A
4.3.8 - Data collection
The data was collected by the external organisation Fivium through their eCase correspondence management software.
4.3.9 - Data cleaning
N/A
4.3.10 - Data sharing agreements
There is no data sharing agreement between Fivium and HM Treasury relating to the Correspondence Triage Automation Tool. No data is provided to Fivium from the Triage Tool. Fivium data is only used as training data.
4.3.11 - Data access and storage
Data is stored in an S3 bucket on an Azure blob storage, accessible only to those who are working on the project. This is secured using pre-approved credentials and requires access via a white-listed IP address.
Tier 2 - Risks, Mitigations and Impact Assessments
5.1 - Impact assessment
N/A - Initial calculations have been done on the cost-saving achieved from using the tool versus human sorting of emails. The tool is currently in the initial phase of implementation. No DPIA has been undertaken at this stage.
5.2 - Risks and mitigations
-
Possible misdirection of emails to wrong person. The overlap in subject matter among HMT groups can occasionally result in emails being misdirected to the wrong group. To mitigate this risk, the hub coordinator is able to swiftly recognise subjects that fall outside the scope of their group and can redirect correspondence to the appropriate party.
-
Inadequate standard lines proposed. Before a response is drafted, correspondence officers read through all lines proposed by the tool and only include where appropriate. In addition, they are trained to have a familiarity with standard lines, such that they are able to identify the appropriate standard line if it is not flagged by the tool. This ensures all responses are thoroughly evaluated before they are sent to the public.