NICE: NORMA (NICE-ONS Recommendation Matching Algorithm)
NORMA is an internal tool that allows staff at NICE to search NICE published guidance and recommendations quickly by undertaking a simple keyword or recommendation search.
Tier 1 Information
1 - Name
NORMA (NICE ONS Recommendation Matching Algorithm)
2 - Description
NORMA is an internal tool that allows staff at the National Institute for Health and Care Excellence (NICE) to search NICE published guidance and recommendations quickly by undertaking a simple keyword or recommendation search.
This tool is utilised to expedite the checking and cross referencing process of existing guidance during a guideline review, significantly reducing the time required - what once took up to a week can now be completed much faster for each guideline search. This tool also helps ensure that NICE recommendations are consistent across various guidelines, and to respond to external queries as quickly as possible by reducing the resource and time extensive documentation searches.
3 - Website URL
The GitHub repositories are private.
This tool is described at the partner website: https://datasciencecampus.ons.gov.uk/supporting-healthcare-guidance-recommendations-using-natural-language-processing/
4 - Contact email
Tier 2 - Owner and Responsibility
1.1 - Organisation or department
National Institute for Health and Care Excellence
1.2 - Team
Surveillance
1.3 - Senior responsible owner
Head of Prioritisation and Surveillance
1.4 - External supplier involvement
Yes
1.4.1 - External supplier
Office for National Statistics (ONS)
1.4.2 - Companies House Number
N/A
1.4.3 - External supplier role
The ONS team worked with NICE to implement the Natural Language Processing (NLP) solution. The solution has since been transferred fully to NICE, and managed and maintained by NICE staff.
1.4.4 - Procurement procedure type
NORMA was developed through a collaboration between NICE and the Office for National Statistics (ONS) Data Science Campus (DSC) as part of the DSC’s objectives to deliver data science projects for public good, or data science skills, assess non-traditional data sources and develop data science methods to provide insight for decision-makers. The services of the DSC were not procured, but provided to NICE without charge in the spirit of collaboration.
1.4.5 - Data access terms
The ONS DSC were given full and unrestricted access to all publicly available NICE publications via the NICE syndication API. There were no terms or conditions placed on their use of the data.
Tier 2 - Description and Rationale
2.1 - Detailed description
NORMA is a tool designed to identify existing NICE guidance by a human inputting a search request that reviews all NICE guidance and provides responses based on similar keywords or a specific recommendation. NORMA was built by undertaking multiple actions. The first action was to build a library of content to search against by extracting recommendations from unstructured guideline texts that are on the NICE website; done using rule-based pattern matching, and organising them into a structured, searchable data set. The second action is extracting the recommendations from the guidance, these are validated by using a test data set consisting of a manually curated set of recommendations corresponding to various guideline topics. The model then extracts these recommendations and transforms them into vectors using a vector embedding model known as Bio-embedding-intrinsic (https://www.nature.com/articles/s41597-019-0055-0)) so they can be easily searched and listed.
This means when a user searches NORMA by entering a keyword search or a recommendation search, the average vector of the input keywords or nouns in the recommendation is calculated using the embedding model mentioned above and compared to the stored NICE recommendation vectors using cosine similarity. The recommendations with the highest similarity scores are returned to the user.
2.2 - Scope
In NICE, the surveillance team is responsible for assessing whether any new evidence impacts on the existing guideline recommendations. With over 300 guidelines, each containing hundreds of recommendations as well as other textual information, this is no small task. One key element of the guideline review process is the identification of overlapping content. NICE needs to make sure that it is consistent in its recommendations and there are no contradictions between guidelines. Checking related guidelines is a manual, time-consuming task as content is stored on the website. This tool was developed to assist staff in undertaking checks against pre-existing guidelines and speed up the review process- in ensuring that NICE recommendations are consistent across guidelines, to support guideline scoping activity and to respond to external queries.
2.3 - Benefit
This tool is utilised to;
Expedite checks during the guideline review process.
Ensuring that NICE recommendations are consistent across guidelines.
To support guideline scoping activity.
Speed up responding to external queries.
Without this tool the manual process can take up to a week of an analyst’s time for reviewing any given guideline.
2.4 - Previous process
Prior to the tool being in place when staff undertook the guideline review process, staff would manually check the published NICE recommendations guidance for any overlapping content, to ensure consistency across NICE recommendations. Doing this manually can take up to a week of an analyst’s time for any given guideline. Although a keyword search within the NICE website was available, this returned webpages rather than specific recommendations. Analysts needed to read through or use a “browser find” function to search to find the specific recommendations of interest. They also needed to manually record their findings.
2.5 - Alternatives considered
During the development of the tool several models were tested such as: CountVectors (TFIDF) Word2vec fastText bioword2vec ELMO BERT The model (Bio-embedding-intrinsyc) was chosen because of its superior performance on the validation dataset.
Tier 2 - Decision making Process
3.1 - Process integration
The tool is available as a standalone, browser-accessible resource for internal use at NICE. NICE staff can utilise it to identify similar recommendations for various business purposes, such as identifying recommendations that require updates or standing down, checking for inconsistencies or overlaps across different guidelines, addressing safety issues (as demonstrated in the example below), identifying gaps in the current portfolio, to find related guidance that may need revision based on new medicine alerts received globally etc.
Example scenario: The Surveillance team receive a safety alert about a specific intervention or medicine that requires the surveillance team to act and update recommendations guidance. The surveillance team will then load up NORMA to find all existing recommendations guidance that contain the related intervention. Once the team have the intervention guidance needed they will undertake an assessment as to the impact of the safety alert. Once impact judgment is made, the required changes (proposed by the analyst who did the assessment) will get signed off by the centre director. Then it will be sent to publishing team to make the changes on the NICE website.
3.2 - Provided information
After conducting a search, the tool provides users with a list of recommendations, each accompanied by a similarity score ranging from 0 to 1, indicating how closely the results match their search input, along with the details of the guideline document the recommendation appears in. The user can also specify how many matching recommendations they would like to see. The tool will then display the corresponding number of recommendations as defined by the user. The results are presented directly in the browser window.
3.3 - Frequency and scale of usage
This tool is now embedded and used in the day to day work of Surveillance team (7 technical staff) to find relevant guidance to update.
3.4 - Human decisions and review
The tool makes the decision about which recommendations contain certain text, all other decisions (e.g. whether the recommendations need updating or standing down, assessing inconsistencies or overlaps, addressing gaps in NICE recommendations, deciding on the course of action based on the safety alert etc.) are made by NICE staff. On the search results returned by NORMA, NICE staff undertake a sense checking to ensure it is what they are looking for, if not they will undertake another search using different phrases and terms.
3.5 - Required training
There is a help manual that accompanies the tool and is also embedded with the tool. The help manual explains the different search approaches and best practice.
3.6 - Appeals and review
The decision made by the tool does not directly affect the public. It’s used to improve the efficiency of our internal checks. The information provided by NORMA is supplemented with institutional and historical knowledge from human experts. Any operational discrepancy are reported to NICE’s Digital team who maintain the tool.
Tier 2 - Tool Specification
4.1.1 - System architecture
NORMA is a tool designed to identify NICE recommendations that are similar to a given set of keywords or a specific recommendation. In its first stage, NORMA extracts recommendations from unstructured guideline texts using rule-based pattern matching, and organises them in a structured format. These extracted recommendations are then transformed into vectors using a vector embedding model known as Bio-embedding-intrinsic (https://www.nature.com/articles/s41597-019-0055-0))
For user-provided keywords or recommendations, the average vector of the input keywords or nouns in the recommendation is calculated using the same embedding model and compared to the stored NICE recommendation vectors using cosine similarity. The recommendations with the highest similarity scores are returned to the user.
URL reference: https://datasciencecampus.ons.gov.uk/supporting-healthcare-guidance-recommendations-using-natural-language-processing/
4.1.2 - Phase
Production
4.1.3 - Maintenance
The data import functionality is changed when new guideline updates break the current parsing logic, as required. This ensures the tool is updated when required to accommodate the latest guidance products. No changes have been made or will be made to the model or front-end elements of the application.
4.1.4 - Models
Rule based recommendation extractor; Bio-embedding-intrinsyc model; Cosine similarity algorithm.
Tier 2 - Model Specification
4.2.1 - Model name
Bio-embedding-intrinsyc; https://github.com/ncbi-nlp/BioWordVec
4.2.2 - Model version
Bio-embedding-intrinsyc 1.0.0 - The model has not been updated since its implementation. The code which does the parsing has undergone changes, and its version control number is 1.0.26.
4.2.3 - Model task
This model is used to convert the recommendation texts and user input into vector embeddings.
4.2.4 - Model input
- Extracted recommendation texts
- User given keywords/recommendation
4.2.5 - Model output
- Average vector of vectors corresponding to nouns in the recommendation text - a list of such vectors, one each for each of the recommendations in the list.
- Average vector of vectors of keywords/nouns in the recommendation text
4.2.6 - Model architecture
Pre-trained subword embedding model. Full details can be found in https://www.nature.com/articles/s41597-019-0055-0
4.2.7 - Model performance
The model’s performance was evaluated to determine the most suitable pre-trained model for the task. Full details of the testing and performance metrics can be found here: https://datasciencecampus.ons.gov.uk/supporting-healthcare-guidance-recommendations-using-natural-language-processing/
4.2.8 - Datasets
NICE datasets were only used for testing purposes. A curated dataset of 100 queries and recommendations corresponding were used for testing various available models and to select the model with the best performance.
4.2.9 - Dataset purposes
To select the best pre-trained model for creating the embeddings
Tier 2 - Data Specification
4.3.1 - Source data name
Surveillance dataset
4.3.2 - Data modality
Text
4.3.3 - Data description
Dataset consists of previous surveillance queries and their associated recommendations.
4.3.4 - Data quantities
Dataset was not used for training/fine-tuning. The dataset was used to analyse performance and review outputs of various pre-trained models and to make the selection.
4.3.5 - Sensitive attributes
None
4.3.6 - Data completeness and representativeness
Dataset was chosen to ensure a wide coverage of all the key NICE guidance products, namely: clinical, public health guidelines, technology appraisals, diagnostic guidance and interventional procedure guidance.
4.3.7 - Source data URL
N/A
4.3.8 - Data collection
The purpose of this dataset was to compare the performance of various pre-trained models, and to select the most appropriate one. The dataset consisted of surveillance queries and recommendations that were identified as still being valid.
4.3.9 - Data cleaning
Once the dataset was processed by the selected model, the data output was reviewed to find any valid recommendations identified by the model, that were not present in the original list, these were added to the dataset after a final further inspection.
4.3.10 - Data sharing agreements
Terms of use for NICE syndication service and API: https://www.nice.org.uk/Media/Default/About/what-we-do/Syndication/uk-syndication-pilot-and-full-licence.pdf
4.3.11 - Data access and storage
The data used in NORMA, is published guidance recommendations and is fully and openly accessible to the public via the NICE website. This content is also available via NICE’s API to registered clients of the NICE syndication service (see terms in section 2.4.3.10, above). The data is stored in NICE’s publications system. The NORMA is an internal tool available to NICE staff only through NICE login.
Tier 2 - Risks, Mitigations and Impact Assessments
5.1 - Impact assessment
There are no formal impact assessments conducted.
5.2 - Risks and mitigations
Risk: Technical stack chosen by ONS is hard to maintain Mitigation: Future decision on the applications future will be informed by the current work on content structuring
Risk: A small proportion of recommendations are missing from NORMA. NORMA may miss valid recommendations. Mitigation: Manual oversight is needed at present.