Lightweight transformers for clinical natural language processing

Abstract

Specialised pre-trained language models are becoming more frequent in Natural language Processing (NLP) since they can potentially outperform models trained on generic texts. BioBERT (Sanh et al., Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01108, 2019) and BioClinicalBERT (Alsentzer et al., Publicly available clinical bert embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 72–78, 2019) are two examples of such models that have shown promise in medical NLP tasks. Many of these models are overparametrised and resource-intensive, but thanks to techniques like knowledge distillation, it is possible to create smaller versions that perform almost as well as their larger counterparts. In this work, we specifically focus on development of compact language models for processing clinical texts (i.e. progress notes, discharge summaries, etc). We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning, with the number of parameters ranging from $15$ million to $65$ million. These models performed comparably to larger models such as BioBERT and ClinicalBioBERT and significantly outperformed other compact models trained on general or biomedical data. Our extensive evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks, including natural language inference, relation extraction, named entity recognition and sequence classification. To our knowledge, this is the first comprehensive study specifically focused on creating efficient and compact transformers for clinical NLP tasks. The models and code used in this study can be found on our Huggingface profile at https://huggingface.co/nlpie and Github page at https://github.com/nlpie-research/Lightweight-Clinical-Transformers, respectively, promoting reproducibility of our results.

1. Introduction

Large language models pre-trained on generic texts serve as the foundation upon which most state-of-the-art Natural language Processing (NLP) models are built. There is ample evidence that, for certain domains and downstream tasks, models that are pre-trained on specialised data outperform baselines that have only relied on generic texts (Sanh et al. 2019; Alsentzer et al. 2019; Beltagy, Lo, and Cohan 2019; Nguyen, Vu, and Nguyen 2020; Chalkidis et al. 2020).

These models, however, are heavy in size and number of parameters, making them unsuitable for devices with limited memory and processing capacity. Furthermore, the rate at which leading technology corporations build these progressively larger and more resource-intensive models is a subject of debate in the Artificial Intelligence (AI) community (Bender et al. 2021), and there is interest in developing methods that would make these tools more accessible by creating smaller and faster versions of them that would run reasonably well on smaller devices (Li et al. 2020; Schick and Schütze 2021). This would allow independent researchers, particularly those from low-income nations, to contribute to the advancement of AI.

From the point of view of energy consumption and environmental impact, developing smaller transformer-based language models can be thought of as a step towards green AI (Schwartz et al. 2020), an approach to developing AI that prioritises sustainable use of computational resources and development of models with minimal carbon footprint (Strubell, Ganesh, and McCallum 2019).

Numerous works exist in the NLP literature with the aim to develop fast, efficient and lightweight versions of larger transformer-based models (Sanh et al. 2019; Jiao et al. 2020; Sun et al. 2020; Dehghani et al. 2021). However, there are comparatively fewer compact models developed for special domains like law, biology and medicine (Ozyurt 2020; Bambroo and Awasthi 2021; Rohanian et al. 2022). The present work focuses on development of efficient lightweight language models specifically developed for clinical NLP tasks. These models can be used to process a range of different clinical texts including patient history, discharge summaries and progress notes. The contributions of this work are as follows:

• • We pre-trained five different compact clinical models using either distillation or continual learning on the MIMIC-III notes dataset.

• • We used three different distillation techniques for training models in varying sizes and architectures.

• • We evaluated our models on named entity recognition (NER), relation extraction (RE) and sequence classification (CLS) on four widely used clinical datasets plus an internal cancer identification dataset.

• • We are the first to focus exclusively on developing compact clinical language models, and we make all of our models publicly available on Huggingface.a

2. Clinical notes in electronic health records (EHR)

Clinical notes are written documents generated by medical practitioners in order to communicate information about a patient treated at a health facility (Rethans, Martin, and Metsemakers 1994). These documents are regarded as ‘unstructured data’. This means that, unlike tabular data that are categorised and quantifiable, clinical notes are irregular, disorganised and not coded using predefined terms that domain experts would all understand (Boulton and Hammersley 2006; Rosenbloom et al. 2011). Clinical notes contain a wide range of information about a patient, ranging from medical history and response to medication to discharge summaries and even billing. Fast Healthcare Interoperability Resources (FHIR)b identifies eight different types of clinical notes, including consultations notes, imaging narratives, laboratory reports and procedure notes (Bender and Sartipi 2013).

Unstructured data constitutes $80$ % of all electronic health records (EHR) data (Kong 2019; Mahbub et al. 2022) and can potentially contain information that is otherwise not present elsewhere in the patient’s EHR (Zhang et al. 2022). They can, therefore, be exploited by computational models to infer more information about a patient or develop predictive models for patient monitoring. There are numerous examples in the literature where clinical notes have been used, sometimes in conjunction with structured data, to develop diagnostic or predictive models. Some examples include adverse drug effects (Dandala, Joopudi, and Devarakonda 2019), (Mahendran and McInnes 2021), self-harm and drug abuse prediction (Obeid et al. 2020; Ridgway et al. 2021), hospitalisation and readmission risk (Huang, Altosaar, and Ranganath 2019; Song et al. 2022), mortality prediction (Si and Roberts 2019; Ye et al. 2020) and automatic phenotype annotation (Zhang et al. 2022).

Unless preprocessed and redacted, clinical notes might contain private information about patients, and models trained on this data are known to be prone to adversarial attacks (Lehman et al. 2021). For this reason, they are yet to be widely used for research and different de-identification methods have been developed to automatically remove personal identifiable information from text documents in order for them to be securely shared with other researchers (Melamud and Shivade 2019; Hartman et al. 2020).

2.1. Biomedical vs clinical texts

There is a distinction in the literature between clinical and biomedical texts and they are understood to be different in terms of their linguistic qualities (Alsentzer et al. 2019). Clinical notes are collected by healthcare professionals when the patient is seen or being treated. They are free-text, without a fixed structure, can contain spelling errors, abbreviations, non-standard grammar, differences in personal style and words and phrases from different languages. These characteristics contribute to the fact that they are still underutilised as a resource (Sanyal, Rubin, and Banerjee 2022).

What is referred to as biomedical texts, on the other hand, are often compilations of scientific texts in the biomedical and life sciences from resources such as PubMed. They are written in a more polished standard style,c and while they do overlap with clinical texts, they are larger in size and easier to process using standard NLP methods.

2.2. Language models for clinical texts

Due to the differences between biomedical and clinical texts (Section 2.1), transformer-based language models that have been only trained on generic and biomedical texts are not always sufficient to capture all the complexities of clinical notes. For this reason, it is common to use pre-trained models as a starting point and either use fine-tuning to adapt them to clinical notes (van Aken et al. 2021; Agnikula Kshatriya et al. 2021) or use continual learning and further pre-train a model like BERT or BioBERT (Lee et al. 2020a) on clinical texts (Si et al. 2019; Alsentzer et al. 2019; Qiu et al. 2020).

2.3. Adapting to new domains via continual learning

Continual learning is a powerful alternative to the standard transfer learning approach which involves pre-training and fine-tuning on a target task. In this paradigm, models can adapt to new domains during the pre-training stage. This linearity resembles biological learning and also alleviates the need for excessive model re-training (Mehta et al. 2021).

The idea here is to adapt the model to new streams of data while retaining knowledge of the previous domains (Parisi et al. 2019). Using this strategy, we may pre-train models that have previously been trained on biomedical texts and expose them to clinical notes and the vocabulary associated with them. While this method is not specifically related to model compression, we can develop lightweight clinical models by using already compressed biomedical models, such as BioMobileBERT, and through continual learning adapt them to the clinical domain. We explore this approach in this work in Section 3.2.

2.4. Language model compression

As discussed in Section 1, issues like overparameterisation, computational overhead and the negative environmental impact of large pre-trained language models have led researchers to develop strategies for compressing these models into smaller, faster, but almost equally performant versions (Sun et al. 2019).

Knowledge distillation (KD) (Hinton et al. 2015) is a well studied and powerful technique that is designed to create such models in a ‘teacher–student’ setup, where the smaller student model learns to mimic its teacher, either task-specifically by using the teacher’s outputs as soft labels or task-agnostically by looking at the outputs of a pre-training objective such as Masked Language Modelling (Devlin et al. 2019). The latter option allows for greater flexibility because the student may then independently be fine-tuned on the target task (Wang et al. 2020).

DistilBERT (Sanh et al. 2019) is a notable example of such an effort in NLP, inspiring a slew of alternative ‘distilled’ versions of commonly used models. DistilGPT2, DistilRoBERTa, DistilBART and DistilT5 are a few examples. More recently, other powerful approaches have also appeared in the literature, some of which will be covered in Section 3.

The efficacy of KD-based compression in specialised domains like biomedical and clinical texts is still understudied. Rohanian et al. (2022) is an example of a work that focuses on development of compact biomedical transformers. To the best of our knowledge, there is no work specifically targeting KD for models trained on clinical texts. As discussed in Section 2.1, these texts contain linguistic features and terminology that differ from generic biomedical texts, necessitating a separate treatment.

3. Methods

In this work, we utilise KD methods (Section 2.4) to train small-sized and efficient language models specialised for processing of clinical texts. First, we use KD approaches to directly extract compact models from the BioClinicalBERT model; second, we employ continual learning to pre-train existing compact biomedical models (e.g. BioDistilBERT and BioMobileBERT) using the MIMIC-III notes dataset (Johnson et al. 2016).

3.1. Compressing clinical models via distillation

In order to distil compact models from BioClinicalBERT, three different KD methods are explored in this work: DistilClinicalBERT, TinyClinicalBERT and ClinicalMiniALBERT (Fig. 3). These methods are described in detail below.

3.1.1. DistilClinicalBERT

This approach follows the distillation process outlined in DistilBERT Sanh et al. (2019) with the aim of aligning the output distributions of the student and teacher models based on the Masked Language Modelling (MLM) objective, as well as aligning their last hidden states. The loss used for this approach is defined as

(1) \begin{align} L(X, Y) &\,= \lambda _{1} L_{\textit{output}}(f_{s}(X), f_{t}(X))\\ &\quad + \lambda _{2}L_{\textit{align}}(h_{s}(X), h_{t}(X)) \nonumber \\ &\quad + \lambda _{3}L_{\textit{mlm}}(f_{s}(X), Y) \nonumber \end{align}

where $X$ is the input to the model, $Y$ represents the MLM labels, $f_{s}(X)$ and $f_{t}(X)$ denote the outputs of the student and teacher models, respectively, $h_{s}(X)$ and $h_{t}(X)$ are the last hidden states of the student and teacher, $L_{\textit{output}}$ is a KL-Divergence loss for aligning the output distributions of the student and teacher, $L_{\textit{align}}$ is a cosine embedding loss for aligning the last hidden states of the student and teacher, $L_{\textit{mlm}}$ represents the original MLM loss and $\lambda _{1}$ to $\lambda _{3}$ are hyperparameters controlling the weighting of each component in the loss function.

The student used in this approach uses six hidden layers, a hidden dimension size of $768$ and an expansion rate of four for the MLP blocks, resulting in $65$ M parameters in total. For initialising the student model, we follow the method as described by Sanh et al. (2019) and Lee et al. (2020b). This involves using the same embedding size as the teacher and borrowing pre-trained weights from a subset of the teacher’s layers.

3.1.2. TinyClinicalBERT

This is a layer-to-layer distillation approach based on TinyBERT (Jiao et al. 2020), which is intended to align the hidden states and attention maps of each layer of the student with a specific layer of the teacher. Because the student network typically uses a smaller hidden dimension size compared to its teacher, an embedding alignment loss is also included. The combined loss in this approach is defined below:

(2) \begin{align} L(X) &\,= \lambda _{0}L_{\textit{embed}}(e_{s}(X), e_{t}(X))\\ &\quad + \sum ^{N}_{l=1} \lambda _{l}L_{\textit{att}}(a_{s}^{l}(X), a_{t}^{g(l)}(X)) \nonumber \\ &\quad + \sum ^{N}_{l=1} \lambda _{l}L_{\textit{hid}}(h_{s}^{l}(X), h_{t}^{g(l)}(X)) \nonumber \\ & \quad + \lambda _{(N+1)}L_{\textit{out}}(f_{s}(X), f_{t}(X)) \nonumber \end{align}

where $N$ represents the number of layers in the student model. The embedding vectors for the student and teacher models before passing to the transformer encoder are represented by $e_s(X)$ and $e_t(X)$ , respectively. For the $i\textit{th}$ layer of both the student and teacher models, the attention maps and hidden states are represented by $a^i_s(X)$ , $a^i_t(X)$ , $h^i_s(X)$ and $h^i_t(X)$ , respectively. The mapping function $g(.)$ is used to determine the corresponding teacher layer index for each student layer and is the same mapping function used in TinyBERT. The mean squared error (MSE) loss $L_{\textit{embed}}$ is used to align the embeddings of the student and teacher models, while the MSE losses $L_{\textit{att}}$ and $L_{\textit{hid}}$ align their attention maps and hidden states, respectively. The cross-entropy loss $L_{\textit{out}}$ aligns their output distributions. Finally, hyperparameters $\lambda _0$ to $\lambda _{(N+1)}$ control the significance of each loss component.

The student model in this approach has four hidden layers, with a hidden dimension of $312$ and an MLP expansion rate of $4$ , totalling $15$ million parameters. Due to the difference in hidden dimensions between the student and teacher models, the student model is initialised with random weights.

3.1.3. ClinicalMiniALBERT

This is another layer-to-layer distillation approach with the difference that the student is not a fully-parameterised transformer, but a recursive one (e.g. ALBERT (Lan et al. 2019)). We follow the same distillation procedure introduced in MiniALBERT (Nouriborji et al. 2022), which is similar to Equation (2). The recursive student model in this method uses cross-layer parameter sharing and embedding factorisation in order to reduce the number of parameters and employs bottleneck adapters for layer-wise adaptation. Its architecture features a hidden dimension of $768$ , an MLP expansion rate of $4$ and an embedding size of $312$ , which add up to a total of $18$ million parameters. Similar to TinyClinicalBERT (Section 3.1.2), the student model is initialised randomly.

3.2. Adapting compressed models to clinical tasks via continual learning

We investigate an alternative method for compressing clinical models through continual learning. In this approach, already compressed biomedical models are further specialised by pre-training on a collection of clinical texts using the MLM objective. Two different models, namely, ClinicalDistilBERT and ClinicalMobileBERT, are developed in this fashion.

To obtain ClinicalDistilBERT, we use the pre-trained BioDistilBERT (Rohanian et al. 2022) model and train it further for three epochs on MIMIC-III. This model has the same architecture as DistilClinicalBERT (as described in Section 3.1.1). ClinicalMobileBERT, on the other hand, is based on the pre-trained BioMobileBERT (Rohanian et al. 2022) model and is also further trained on MIMIC-III for three epochs. This model has a unique architecture that allows it to have a depth of up to $24$ hidden layers while having only $25$ million parameters.

4. Datasets and tasks

This section discusses the NLP tasks and datasets used in this work. We briefly explain the goals and the nature of each task, followed by information on the datasets used to evaluate the proposed models.

4.1. Tasks

We explored four prominent tasks in clinical NLP: NER, Relation Extraction (RE), natural language inference (NLI) and lassification (CLS). Below, we provide definitions and concrete examples from each task to illustrate their objectives and characteristics.

4.1.1. Named entity recognition (NER)

NER is the task of automatically processing a text and identifying named entities, such as persons, organisations, locations and medical terms. For example, in the sentence ‘The patient was diagnosed with heart disease by Dr. Johnson at JR Hospital’, NER would identify ‘patient’, ‘heart disease’, ‘Dr. Johnson’ and ‘JR Hospital’ as named entities and classify them as ‘Person’, ‘Disease’, ‘Person’ and ‘Hospital’, respectively.

4.1.2. Relation extraction (RE)

RE is the task of recognising and extracting links between entities such as genes, proteins, diseases, treatments, tests and medical conditions. For example, in the sentence ‘The EGFR gene has been associated with increased risk of lung cancer’, the RE system may recognise the association between the EGFR gene and lung cancer as ‘connected to increased risk’.

4.1.3. Natural language inference (NLI)

In natural language inference (NLI), the goal is to determine the connection between two texts, such as a premise and a hypothesis. The connection may be defined as entailment, contradiction or neutral. For example, if given the premise ‘The patient is diagnosed with influenza’ and the hypothesis ‘The patient is being treated for bacterial infection’, the NLI system would find that the connection is ‘contradiction’. This task helps improve understanding of the connections between biomedical concepts in language.

4.1.4. Sequence classification (CLS)

CLS is the task of assigning class labels to word sequences in a biomedical text, such as sentences or paragraphs. The aim is to correctly predict the sequence’s class label based on the contextual information provided in the text. For example, given the text ‘Patient has high fever and severe headache’, the system may predict the class label ‘symptoms of an illness’.

When the class labels contain assertion modifiers (negation, uncertainty, hypothetical, conditional, etc.) and reflect degrees of certainty, the task is referred to as assertion detection (AD) (Chen 2019), which can be regarded as a subtask of CLS. For example, in the statement ‘Patient has heightened sensitivity to light and mild headache, which may indicate migraine’, the AD system would predict the class label ‘uncertain’ or ‘possible’ based on the context.

4.2. Datasets

We evaluate all of our models on four publicly available datasets, namely, MedNLI, i2b2-2010, i2b2-2012, i2b2-2014, and one internal dataset named ISARIC Clinical Notes (ICN).

4.2.1. MedNLI

MedNLI Romanov and Shivade (2018) is a NLI task designed for medical texts, in which two sentences are given to the model and the model should predict one of the entailments, contradiction or neutral labels as the relation of the two given sentences, as shown in Table 1.

Table 1. Training samples from the MedNLI dataset

4.2.2. i2b2 datasets

i2b2-2010 (Uzuner et al. 2011) is a medical relation extraction dataset,d in which the model is required to output the relation between two concepts in a given text. The relations are between ‘medical problems and treatments’, ‘medical problems and tests’ and ‘medical problems and other medical problems’. In total, this dataset uses nine labels which are as follows:

1. 1. Treatment improves medical problem (TrIP)

2. 2. Treatment worsens medical problem (TrWP)

3. 3. Treatment causes medical problem (TrCP)

4. 4. Treatment is administered for medical problem (TrAP)

5. 5. Treatment is not administered because of medical problem (TrNAP)

6. 6. Test reveals medical problem (TeRP)

7. 7. Test conducted to investigate medical problem (TeCP)

8. 8. Medical problem indicates medical problem (PIP)

9. 9. No relations

For fine-tuning our models on this dataset, we follow the same pre-processing used in the BLUE benchmark, which models the relation extraction task as a sentence classification by replacing the concepts with certain tags, as shown in Table 2

Table 2. i2b2-2010 samples taken from the dataset’s guideline Uzuner et al. (2011). The concept pairs for which a relationship should be predicted are displayed in boldface. Following the pre-processing used in the BLUE benchmark, the concepts are replaced with tags and then passed to the model as shown in the second column

i2b2-2012 (Sun, Rumshisky, and Uzuner 2013) is a temporal relation extraction dataset that contains $310$ discharge summaries from Partners Healthcare and the Beth Israel Deaconess Medical Center. It contains inline annotations for each discharge summary in four categories: clinical concepts, clinical departments, evidentials and occurrences. In our experiments, it is used as an NER dataset with the following entity labels:

1. 1. Medical Problem (PR)

2. 2. Medical Treatment (TR)

3. 3. Medical Test (TE)

4. 4. Clinical Department (CD)

5. 5. Evidental (EV)

6. 6. Occurrence (OC)

7. 7. None (NO)

Some samples from the training dataset are provided in Figure 1.

Figure 1. Samples from the i2b2-2012 dataset.

i2b2-2014 (Stubbs, Kotfila, and Uzuner 2015) consists of two sub-tasks: De-identification and heart disease risk factors identification. In our experiments, we focus on the de-identification task in which the goal is to remove protected health information (PHI) from the clinical notes. The data in this task contain over $1300$ patient records and has inline annotations for PHIs in each note. Similar to i2b2-2012, this task is also framed as NER with $22$ labels. Figure 2 shows some examples taken from the training subset of the dataset.

Figure 2. Samples from i2b2-2014. Names have been anonymised for privacy. Labels are ‘PA’ for Patient’, ‘PR’ for ‘Professional’, ‘OR’ for ‘Organisation’ and ‘DO’ for ‘Doctor’. See Appendix A for the complete list of labels.

4.2.3. ISARIC clinical notes (ICN)

The ISARIC COVID-19 Clinical Databasee consists of data from patients hospitalised with COVID-19 who are enrolled in an ISARIC Partner clinical study (Garcia-Gallo et al. 2022). The data (which are standardised to CDISC STDM formatf) include hospital admission and discharge records, signs and symptoms, comorbidities, vital signs, treatments and outcomes. Non-prespecified terms related to one of five categories; medical history, complications, signs and symptoms, new diagnosis at follow-up or category not recorded. The non-prespecified terms may consist of one or multiple diagnoses or clinical events within the same category.

In December 2021, when the initial stratified sample of non-prespecified (free text) medical terms from the ISARIC COVID-19 Clinical Database was extracted, the database comprised of data from over 708,231 participants. The sample was formed of 125,381 non-prespecified terms and all five of the aforementioned categories were represented. We have released a relevant subset of this dataset along with this work and a copy of ICN, with necessary modifications for patient confidentiality, is available at https://github.com/nlpie-research/Lightweight-Clinical-Transformers/tree/main/ICN.

For the experiments in this work, each instance of the data consists of non-prespecified (free text) terms describing clinical and adverse events along with a classification label from a number of possible choices. An annotator with clinical training used the following three labels to annotate the free-text notes:

1. 1. Malignancy

2. 2. No Malignancy

3. 3. Possible Malignancy (Ambiguous)

This annotation scheme is associated with the task of AD van Aken et al. (2021) as explained in Section 4.1.4. We refer to this annotated portion as ISARIC Clinical Notes (ICN) cancer classification dataset. Table 3 contains a few examples from ICN.

Table 3. Some sample clinical notes along with their annotation from the ICN dataset

5. Experimental setup

5.1. Pre-training details and hyperparameters

We pre-train all of our models on the MIMIC-III dataset for a total of three epochs using either the MLM objective or Knowledge Distillation. We follow the same pre-processing used in Alsentzer et al. (2019) for MIMIC and use the BERT tokeniser from the Huggingface with a max length of $256$ tokens. The details of the hyperparameters used for pre-training and fine-tuning our models are available in Tables 7 and 8.

5.2. Results

We evaluated the proposed models and the baselines on five datasets: MedNLI, i2b2-2010, i2b2-2012, i2b2-2014 and ICN. All reported results are median of three runs with different seeds. As shown in Table 4, compact clinical models significantly outperform their general and biomedical baselines and achieve competitive results against BioBERT-v1.1 and BioClinicalBERT.g ClinicalDistilBERT and ClinicalMobileBERT, which are trained using continual learning, obtain the best average results among all compact models (Table 4). ClinicalMiniALBERT outperforms both DistilClincialBERT and TinyClinicalBERT in terms of average results among our distilled models.

Table 4. The results of the baselines (above the double line) and our pre-trained models on clinical downstream tasks. The metrics used for reporting scores are accuracy for the MedNLI, micro-averaged F1 for i2b2-2010 (RE), macro-averaged F1 for ICN, and Exact F1 for the others. Bold numbers denote the best performance and underlined numbers denote the second-best performance

6. Discussion and analysis

6.1. Effect of different initialisations

Following the work of Alsentzer et al. (2019), we explore the effect of different initialisations for our continually learned models, as shown in Table 5. We find that initialising ClinicalDistilBERT with the trained weights from a pre-existing biomedical model significantly improved the model’s average performance, particularly on the MedNLI dataset. However, we discovered that initialising the ClinicalMobileBERT with a similar biomedical checkpoint did not result in a significant performance boost.

Table 5. The effect of different initialisations on the continual learning of compact models

6.2. Performance evaluation through latency and GMACs

Latency and Giga Multiply-Accumulate Operations per second (GMACs) are two important metrics used to evaluate the efficiency of machine learning models. Latency measures the amount of time it takes for a model to process a single input and produce an output and is typically measured in milliseconds. GMACs, on the other hand, stands for Giga Multiply-Accumulate Operations and is a measure of how many computational operations a model needs to perform its tasks. GMACs are expressed in terms of billions of operations and provides a way of quantifying the computational resources required by a machine learning model.

The results of the latency and GMACs of the models, as well as the model sizes, are presented in Table 6. These measurements were conducted using a V100 32G GPU. The results show that there is a trade-off between accuracy and efficiency of the models. The ClinicalBioBERT model, which is listed only as representative of a class of larger-sized models (110m parameters, as listed in Table 4), has the best performance on the test sets but has the highest latency and GMACs, making it less suitable for deployment in resource-constrained environments. On the other hand, the TinyClinicalBERT model has the lowest latency, GMACs and size, but its performance may not be as good as that of ClinicalBioBERT.

Table 6. Comparing the efficiency of the proposed models with ClinicalBioBERT. $\downarrow$ denotes that less is better for that particular metric

The DistilClinicalBERT, ClinicalDistilBERT, ClinicalMobileBERT and ClinicalMiniALBERT models offer a good balance between performance and efficiency with relatively lower latency, GMACs and smaller sizes compared to the ClinicalBioBERT model. The BioClinicalBERT and BioBERT-v1.1 models, with 110 million parameters, offer the highest performance but are also the most computationally intensive.

In a real-world setting, the choice of the appropriate model should depend on the specific requirements of the application, such as the required accuracy, computational resources and memory constraints. Based on the results presented in Table 4, the DistilClinicalBERT, ClinicalDistilBERT, ClinicalMobileBERT and ClinicalMiniALBERT models with $65$ million, $25$ million and $18$ million parameters, respectively, provide a good balance between performance, latency, GMACs and model size.

6.3. ICN error analysis

6.3.1. Preparation of the error analysis subset

To perform error analysis, we chose three of the best-performing models, namely, BioBERT, BioClinicalBERT and our proposed ClinicalDistilBERT. In order to evaluate the models on a truly unseen dataset for this analysis, we selected the internal ICN dataset (Section 4.2.3). It consists of approximately 125,000 clinical notes, with $6000$ of them having been labelled by clinicians from the ISARIC consortium. We fine-tuned the pre-trained models on the labelled section of the ICN and then used the resulting models to predict labels for all 125,000 samples. The samples in which at least two of the models disagree on the predicted label were identified as corner cases, resulting in approximately $1500$ clinical notes.

To perform the error analysis, these corner cases were annotated by a clinician and the outputs of the three fine-tuned models were analysed and compared with the expert human annotations. More information about the specifics of this adjudication is provided in the Appendix C. Figure 4 provides the confusion matrices for performance of the models both on the test set and on the corner cases. Based on the information from the confusion matrices, ClinicalDistilBERT performed better than the rest of the models on these cases. BioBERT, on the other hand, fared comparatively poorly, which can be attributed to its lack of clinical domain knowledge. In the following section, we present the analysis and observations of the human expert annotator about the outputs of each model on the corner cases and investigate if there are any common mistakes or recurring patterns that have caused confusion for any of the models.

6.3.2. Patterns observed in error analysis

Interpretation of ‘ca’ Abbreviation: In the portion of ICN on which ClinicalDistilBERT was trained, the abbreviation ‘ca’ often refers to cancer. However, these two letters could also be used in other contexts; one such example was a free text term containing ‘low ca+’. It was assumed this free text was referring to low calcium levels, therefore a ‘No Malignancy’ label was assigned by the human annotator. This was in direct contrast to the ‘malignant neoplasm’ label output of ClinicalDistilBERT. On this occasion, description of the level (i.e. low) preceding ‘ca’ and the addition sign succeeding this abbreviation indicated to human annotator that this term refers to an ion outside an expected range. It could, therefore, be reasonably assumed this term refers to a calcium measurement rather than cancer. This example shows how ClinicalDistilBERT has the potential to generate more reliable results if it is further developed to more accurately process contextual information in cases where free texts comprise abbreviations with multiple interpretations.

Mislabelling of Benign Terms (Adenoma, Hemangioma, Angioma): A frequently occurring inconsistency between the labels assigned by a human annotator and ClinicalDistilBERT related to certain free text terms containing the words ‘adenoma’, ‘hemangioma’ or ‘angioma’. Some examples of false positives in Clinical DistilBERT related to the term ‘adenoma’ are as follows: ‘bilateral adenomas’, ‘benign adrenal adenoma’, ‘benign ovarian cystadenoma’ and ‘fibroadenoma of right breast’. In all of these cases, the human annotator decided the note was ‘No Malignancy’ but the model labelled it as ‘Malignancy’. It is particularly interesting that ClinicalDistilBERT incorrectly labelled these, given the fact the some samples start with the word ’benign’. The model may have been misled due to the similarities between the terms ‘adenocarcinoma’ (a type of malignancy) and ‘adenoma’ (a benign neoplasm).

We discovered a pattern of discrepancy between ClinicalDistilBERT labels and human annotator labels in samples containing words such as ‘hemangioma’ or ‘angioma’. For example, the model labelled ‘liver hemangioma’ and ‘right frontal cavernous angioma’ as ‘Possible Malignancy (Ambiguous)’ but the human annotator deemed these terms as ‘No Malignancy’. This may have occurred as the structure of these words is broadly similar to a number of conditions, such as ‘astrocytoma’ and ‘meningioma’, for which it is difficult to decipher the benign or malignant nature of the free text term; these terms were therefore previously seen by the model in its training samples associated with possible malignancy. More specifically, the aforementioned diagnoses with human and model annotation disparity end in ‘-oma’; free text terms ending in ‘-oma’ may often refer to a malignancy, but this is not universally correct. The model may have identified these samples incorrectly due to spelling similarities to malignancy terms encountered during training, but it has not yet developed the ability to differentiate between them consistently.

Acronym Misinterpretation (CLL, ALL, NHL): An analysis of label discrepancies among the three models revealed a pattern of incorrect labelling of certain acronyms by BioBERT and BioClinicalBERT. The models frequently mislabelled three-letter acronyms such as ‘CLL’, ‘ALL’ and ‘NHL’ as ‘No Malignancy’, while both ClinicalDistilBERT and human annotation identified them as malignant conditions. These acronyms can commonly be understood to refer to chronic lymphocytic leukaemia, acute lymphoblastic leukaemia and non-Hodgkin lymphoma, respectively. Despite the fact that these acronyms were included in the ‘Malignancy’ training samples, BioBERT and BioClinicalBERT often labelled free text terms containing these acronyms as ‘No Malignancy’. On the other hand, these models sometimes labelled certain acronyms such ‘gbs’ as ‘Malignancy’, while human annotation and ClinicalDistilBERT did not identify them as malignant.h

Impact of Text Length on Diagnosis (Term ‘Calculi’): The word ‘calculi’ frequently led to inaccuracies in the diagnosis label of all three models. Out of the 1544 corner cases studied during the error analysis, $33$ contained the term ‘calculi’ and were all annotated as ‘No Malignancy’ during human annotation. ClinicalDistilBERT, however, labelled $17$ of these as malignant. These samples usually contained short free text terms, with $10$ having $3$ or fewer words and none having more than $5$ words. Examples include ‘bladder calculi’, ‘left renal calculi’ and ‘staghorn calculi’. The $16$ samples which were correctly labelled as ‘No Malignancy’ by ClinicalDistilBERT were frequently of a greater length, with $8$ samples containing $4$ – $20$ words. Examples of these samples are: ‘psoriasis/ depression/ renal calculi’, ‘gall bladder + common bile duct (cbd) calculi’ and ‘calculi in gallbladder and common bile duct’. BioBERT and BioClinicalBERT only correctly labelled one free text term containing ‘calculi’ as ‘No Malignancy’.

7. Conclusion and future works

In this work, we have presented five compact clinical transformers that have achieved significant improvement over their baselines on a variety of clinical text-mining tasks. To train our models, we utilised both continual learning and knowledge distillation techniques. For continual learning, we employed the BioDistilBERT and BioMobileBERT models, which are compact models derived using standard techniques and trained on large biomedical data. Our experiments showed that the average performance of these models can increase by up to $2\%$ after training on the MIMIC-III clinical notes dataset, and they even outperformed larger baselines on the i2b2-2010 and ICN datasets (Table 4). In order to determine the best approach for knowledge distillation in clinical NLP, we explored a range of different methods including standard (Sanh et al. 2019), layer-to-layer (Jiao et al. 2020) and recursive distillation (Nouriborji et al. 2022).

Moreover, to confirm the efficacy of our methods on an unseen private dataset, we evaluated the performance of the top models on the ICN dataset by looking at the corner cases in the test set where at least two of the models disagreed and then asked an expert annotator with clinical training to adjudicate the corner cases. In this way, we managed to further assess the models on more complicated samples and provided a more in-depth analysis of where the models tend to fail and what recurring themes exist in the more challenging cases.

We also evaluated the models in terms of efficiency criteria such as latency and GMACs and compared the proposed models with BioClinicalBERT. We subsequently provided guidance on selecting the optimal model based on performance and efficiency trade-offs. We hope that by making our lightweight models public, we will make clinical text-mining methods more accessible to hospitals and academics who may not have access to GPU clusters or specialised hardware, particularly those in developing countries. The top-performing models produced in this study will be integrated into the data curation pipeline under development by ISARIC and Global.health,i thereby facilitating the rapid aggregation and analysis of global data for outbreak response.

The work presented here has some limitations, however. Currently, our experiments are limited to datasets in English and it remains to be seen how the models would perform on datasets in other languages. We have also not tested our models on the task of connecting named entities to medical knowledge-bases such as SNOMED CT or ICD-10, which is an important task in clinical NLP. In future work, we aim to extend our research to more tasks and languages in order to address these limitations.

Another potential avenue for future work is the integration of information from other modalities such as images or electronic health records containing tabular data, including clinical laboratory test results and radiology images. This would allow us to train and evaluate multi-modal architectures, such as CLIP (Radford et al. 2021), and explore their utility for clinical NLP tasks.