DSIT: Succession Select
A search tool to support identification of current Senior Civil Servants with career profiles that match requirements for senior digital role vacancies.
Tier 1 Information
Name
Succession Select
Description
Succession Select is an enhanced search assistant powered by a large language model (LLM). It is designed to search through an authorised database containing the career profiles of current senior civil servants from the Government Digital and Data profession and identify a list of potential candidates whose profiles may match the requirements of Senior Civil Servant (SCS) digital role vacancies.
The filtering and matching capability automates the process of human talent acquisition specialists who otherwise would have to manually search through a large database of career profile data. The returned list of potential candidates is reviewed by humans for further consideration and evaluation when making the final selection on candidates.
Website URL
N/A
Contact email
Tier 2 - Owner and Responsibility
1.1 - Organisation or department
The Central Data and Data Office (CDDO)
1.2 - Team
Talent Acquisition Team
1.3 - Senior responsible owner
Deputy Director, Digital Workforce and Capability
1.4 - External supplier involvement
No
Tier 2 - Description and Rationale
2.1 - Detailed description
Succession Select employs a Large Language Model (LLM) to perform a search function to identify Civil Servant career profiles which match the requirements of current vacancies. These identified Senior Civil Servants can then be made aware of the ongoing open role vacancy.
The LLM is first used to generate an ideal career description for the specified digital job vacancy. This generated career description is constructed so that its format matches that of the information collected from the annual Talent Survey of Senior Civil Servants and includes typical work experience job titles and a example list of skills for that role based on the LLM’s own trained dataset for tech roles. Then a second LLM call compares this ideal career description against the pseudonymised career information of senior civil servants in the database. The model considers various factors including career history, skills, grade and candidate’s career aspirations, to evaluate suitability when returning a list of potential candidates.
Finally, the tool returns a list of the best matching anonymised candidates with the actual candidate names appended for human talent specialists to review.
2.2 - Scope
Succession Select firstly makes use of a large language model (LLM) to generate an ideal career description for the specified digital job vacancy. It produces a description in the same terms as the information held in the database (which was collected predominantly via drop-down questions), including typical previous job titles and a list of exemplar skills for that role derived from standard civil servant job descriptions taken from the GRID datasource.
Subsequently, the LLM compares this ideal career description against the pseudonymised career profiles of senior civil servants in the database. The model considers various factors including career history, skills, and grade to evaluate suitability.
Finally, the tool returns a list of the best matching anonymised profiles with the actual candidate names appended for human talent specialists to review. This list of potential candidates is then considered by the human talent acquisition specialists.
2.3 - Benefit
It is estimated that an initial manual search of candidate information using a key wordsearch can take up to 4 hours, where as the Succession Select tool provides a similar search functionality for an initial search in a minute or two.
2.4 - Previous process
On receipt of the vacancy role name the Talent Acquisition Team would need to perform a search of the career information to identify those candidates who meet the required criteria. In order to do this they would perform a keyword search, selecting appropriate terms relating to the role to find those profiles that used those terms.
2.5 - Alternatives considered
Standard machine learning natural language processing was considered, but its performance was vastly inferior to the capability of current LLMs. It also doesn’t allow for creation of pseudo-profiles of both seen and unseen job roles, whereas an LLM will have that within its training data.
Tier 2 - Decision making Process
3.1 - Process integration
Succession Select is designed to be a search tool within the existing human-led process of supporting the career progression of civil servants.
On receipt of the role vacancy details the Talent Acquisition Team would use the Succession Select search tool to determine a list of candidates who had relevant work experience and skills.
The team would then examine these potential candidates in detail, including their grade, career aspirations plans for requirement for next role to produce an ordered shortlist of appropriate candidates together with the rationale for their inclusion.
3.2 - Provided information
The tool initially produces a long list of all the pseudonymised career profiles which have some broad relevance to the particular role vacancy based on a similarity search using vector embeddings.
This long list is then analysed by the LLM to produce a shorter selection containing those profiles which are deemed a good match to the requirement of the job vacancy. Each pseudonymised profile has a unique ID, and using this together with a separate mapping file, the candidate name is reattached to the profile. The user is ultimately presented with the named career profiles for each in the short list with a statement of which features in each profile make them a good match for the vacancy, together with the long list of all candidates with broad relevance to the role. The Talent Acquisition Team review the suggested profiles to produce an ordered shortlist of appropriate candidates together with the rationale for their inclusion. They continue to have access to all career profile data and digital Senior Civil Servants and can refer to these if necessary.
3.3 - Frequency and scale of usage
Succession Select is designed to be used as part of an existing human-led process. This existing process is initiated when government organisations request assistance in identifying current Civil Servants who may be a good fit for openly advertised Civil Service Job vacancies. This tool is used ad-hoc and only when there are low application numbers for open roles.
3.4 - Human decisions and review
The Succession Select tool does not make the final decisions; instead, it facilitates the candidate selection process by providing a long list of potential candidates to human talent specialists for consideration.
The returned list will only include career profiles that have some relevance to the job vacancy being searched for. For example a profile with only experience in Project Management is unlikely to be returned against a search for a Head of Data Science role.
The Talent Acquisition Team are responsible for the onward use of these suggestions, making final decisions based on their experience and detailed evaluation of candidate profiles. Additionally, users are able to rerun the tool with alternative job requirements and have full flexibility to ignore the suggestions or conduct their own manual keyword search of the database to include additional candidates as they wish. The tool outputs a list of candidates with some relevance to the suggested role based on backend data, to also save time if the LLM’s suggested candidates are not acceptable.
3.5 - Required training
-
Tool build: The tool was designed by an experienced team of Senior Data Scientists within the Central Digital and Data Office. To be able to build and maintain a GenAI tool developers would need skills such as: a strong foundation in machine learning, specifically in NLP and LLMs, to understand model integration and prompt engineering. Proficiency in cloud services, particularly AWS, is crucial, including expertise in AWS Lambda for serverless functions, Amazon S3 for data storage, API Gateway for creating and managing APIs, Cognito for authentication, and IAM for secure access management. Robust software engineering skills are necessary, including backend development in languages like Python, API development, version control with Git, testing, debugging, and familiarity with DevOps practices and CI/CD pipelines.
-
Tool users: The tool is used exclusively by authorised, competent individuals within the Talent Acquisition Team, and has an intuitive interface, requiring minimal training. The tool has an associated user guide and technical documentation. Users are taught to use multiple searches per job listing to ensure a thorough search of potential candidates
3.6 - Appeals and review
This is an internal tool, only available for the talent team in CDDO. Succession Select would only be used as one part of an existing human-led process. This is after the user has searched via the system for a particular role vacancy, they can then review the candidate long-list provided by the tool to select suitable candidates and produce an ordered shortlist of appropriate candidates together with the rationale for their inclusion.
HR teams from shortlisted individuals departments are engaged to contact the short listed individual. Any questions, concerns or further requests can be made via their departmental HR team.
Tier 2 - Tool Specification
4.1.1 - System architecture
Succession Select leverages a Retrieval-Augmented Generation (RAG) architecture, securely deployed within an Amazon Web Services (AWS) environment. The system is designed to handle user queries relating to candidate selection in tech roles.
Key components of the architecture include: - Data Ingestion: Data ingestion of the Senior Civil Servant (SCS) career details into the AWS environment is conducted using the open rAPId tool (https://rapid.readthedocs.io/en/latest/)). The data is embedded and transformed into vector embeddings using Amazon Bedrock Knowledge Base.
-
Data Embedding and Storage: The documents are chunked, and their embeddings are generated using Amazon Titan Text Embeddings v2. These embeddings, which are numerical representations of the text, are stored in an OpenSearch vector database, allowing for efficient retrieval based on semantic similarity.
-
Generative AI Integration: The retrieved documents are processed using Anthropic’s Claude, a state-of-the-art LLM integrated via AWS Bedrock. Claude summarises the retrieved information and generates a user-friendly response, which is then presented to the user.
Streamlit is used to provide the user interface.
4.1.2 - Phase
Beta/Pilot
4.1.3 - Maintenance
Succession Select is still in its pilot phase, but will undergo periodic technical review once operational. During use, its users are able to provide feedback, which will drive future updates.
A data pipeline has been established to allow the Talent Acquisition Team to update, add or remove Senior Civil Servant career profiles into the architecture, and the tool has an automated process to refresh its answers/knowledge after new information has been incorporated.
4.1.4 - Models
Anthropic Claude Sonnet 3.5 - Large Language Model Amazon Titan Text Embeddings v2
Tier 2 - Model Specification
4.2.1 - Model name
Anthropic Claude Sonnet
4.2.2 - Model version
3.5
4.2.3 - Model task
- Use user-specified digital role vacancy to generate idealised career description.
- Use logical reasoning with a stated set of rules via prompt engineering to identify pseudonymised candidate descriptions which best match the requirements of the job vacancy.
4.2.4 - Model input
- Career information, including work experience and skills.
- Job title and grade.
- Pseudonymised candidate information and idealised career description
4.2.5 - Model output
- Pseudonymised candidate descriptions.
- List of pseudonymised candidate information.
4.2.6 - Model architecture
Generative pre-trained transformers https://www.anthropic.com/news/claude-3-5-sonnet
4.2.7 - Model performance
The foundational Anthropic models in AWS cannot be fined-tuned. They are ready to use, out-of-the-box models. The way to enhance these models is to provide them with context and rules, so when the response to the user is formulated, the model retrieves the data from the vector database, and alongside its already innate language abilities, it can formulate an appropriate response.
Formal testing undertaken includes: - Factual accuracy testing (100% correct information): Ensuring that returned data from the database for individual candidates was not hallucinated and lined up against the candidate profiles. - Precision evaluation using synthetic profiles (100% correctly identified): Creation of a variety of “fake” profiles to ensure that given the required data, the LLM was consistently able to extract and list those profiles we would expect the LLM to select as a priority.
- Assessment of subjective responses considered acceptable: For several candidate search requests by the talent team, we used the LLM to identify potential candidates. Given the high subjectivity of the use-case, there was overlap but additional requirements (outside the scope of the LLM’s database currently) were considered by the team in making their selections. However, the tool was maintaining its in-built rules around skills/work history/grade/time in role and identified some candidates the talent team had not considered.
Testing was undertaken to ensure that the matching process could not present any gender bias. Initial results found that biased results could be deliberately requested (but not produced), so guardrails were introduced to prevent unethical requests.
4.2.8 - Datasets
Claude’s base model has received large amounts of training data, fine-tuning and configuration to create its base model. The model is not open source, proprietary to Anthropic.
For information on what datasets have been used to train the model please see: https://support.anthropic.com/en/articles/7996885-how-do-you-use-personal-data-in-model-training.
Some examples: - Publicly available information via the Internet - Datasets that we license from third party businesses - Data that our users or crowd workers provide
4.2.9 - Dataset purposes
Claude’s base model has received large amounts of training data, fine-tuning and configuration to create its base model. The model is not open source, proprietary to Anthropic
Tier 2 - Data Specification
4.3.1 - Source data name
- GRID (Government Recruitment Information Database): details of civil service jobs going back to September 2022.
- Smart Survey: results of annual survey of SCS1 and G6 Government Digital and Data professionals.
- Government People Group (GPG) SCS2 data: Provided to talent team by GPG.
- LinkedIn: for those individuals who have given permission, to their Linkedin profiles.
4.3.2 - Data modality
Text
4.3.3 - Data description
- GRID: List of relevant digital and data skills for particular job roles.
- Smart Survey: grade, work experience, time in current role, next move, next move plan, next move requirements, next move priorities, skills and career aspirations.
- LinkedIn: previous work experience and skills where available
4.3.4 - Data quantities
Approximately 500 Senior Civil Servants are currently in the Government Digital and Data profession dataset (~70% G6 and the rest SCS1). The collected information on each candidate is not paragraphs of information, but mainly comprises defined lists of terms taken from drop down menus (e.g. current grade), lists of terms (e.g. previous job titles and skills) or short sentences (e.g. career aspirations).
4.3.5 - Sensitive attributes
The fields used from the data sources are: “grade”, “work experience”, “time in current role”, “next move”, “next move plan”, “next move requirements”, “next move priorities”, “skills”, “career aspirations” and “talent marking”. To reduce potential bias on gender and ethnicity grounds, staff names are removed and replaced with unique identifiers. When the LLM reviews and provides the response, it does not have access to the names, only the unique ids which means that the review is done on the candidates skills and career history. No other personal data is available or used, only name data.
4.3.6 - Data completeness and representativeness
Smart Survey data available on approximately 90% current senior digital and data staff (G6-SCS2) since the baseline Talent Survey is conducted annually (So any staff who have joined, promoted or did not complete the survey would not be included. Other than work experience and career aspiration, this dataset is made up of data collected via dropdown boxes. The underlying GRID datasource is updated every week, and has records going back to 2022. LinkedIn data only available for those who have authorised its inclusion.
4.3.7 - Source data URL
N/A
4.3.8 - Data collection
Smart Survey data obtained through a talent-team-led annual survey for the purpose of identifying skills and work experience as well as career aspirations etc amongst G6 and SCS1 roles.
GRID data is used to complete data points that were not collected or not entered during the smart survey. GPG - this source contains information on SCS2 grades and above. This data is collected for this purpose and is the same questions as the Smart Survey.
Within the Smart Survey users have the opportunity to permit access to LinkedIn data to skip questions (if their profile is public) and is obtained directly from the LinkedIn website for this purpose which includes columns; Work Experience and Skills data is filled using the Linkedin data request. Ad Hoc data - Data collected from SCS3 network for this purpose.
4.3.9 - Data cleaning
Some data cleaning is performed to standardise the raw data: - Standardising job names and using GRID data to identify associated skills - Removing junior Civil Servant grade data - Calculating time in role - Extracting relevant data and structuring the dataset ready for review by the LLM.
The model does not pre-populate any missing data
4.3.10 - Data sharing agreements
Data is shared with the data processor Amazon Web Services and AWS Bedrock to provide data hosting services as part of the Cabinet Office contract with AWS.
4.3.11 - Data access and storage
This Data Science Team in CDDO have access to this data. The data is stored in an S3 bucket, in a restricted AWS account. Only a few specific CDDO members have access to the AWS account, which requires multi-factor authentication.
Tier 2 - Risks, Mitigations and Impact Assessments
5.1 - Impact assessment
A Data Protection Impact Assessment (DPIA) has been completed and approved (review held May 2024). Residual risk considerations: “The technology only provides an enhanced search capability, making it easy to query a database, and supports an existing process” Overall residual risk rating: LOW. No information on gender, ethnicity and diversity is collected or used by the tool. Only name data is available.
5.2 - Risks and mitigations
Potential risks:
- Data breach of personal and sensitive information on 3rd party platform. Control: Review by information security team. Residual risk: LOW
- GDPR Article 22 - Automated decision making. Control: A human will review the output from the tool and will conduct an in depth assessment of appropriateness and fairness. Residual risk: LOW
- Data becoming out of date - the data collected via surveys is not compulsory, meaning if staff don’t update their information each via the Smart Survey they might not be recommended.
- Reinforcement Bias - There are no reinforcement learning techniques taking place from the information that is provided to the algorithm.
- Automation Bias - There is training on the use of the tool with guides for users explaining the tools strengths and weaknesses. No grading or scoring of data/users takes place and is limited to recalling pre-existing information.
- Gender bias - Testing found that male and female records are displayed fairly as expected, and additional guardrails were implemented to prevent unethical requests by the user. Less than 1% of the data provided by staff could have gender inferred (for example by reference to maternity leave or aspiring to support women through use of AI), and much of this would not be provided to the tool. Due to minimal data and it not being relevant to digital job skills, the risk is very low. Furthermore the tool can not learn nor can it be reinforced by profiles that are selected in previous searches.