Few-shot Named Entity Recognition: Definition, Taxonomy and Research Directions

Few-Shot NER (FS-NER) is a sub-field of Few-Shot Learning (FSL), which in turn can be considered as a sub-area in machine learning. We will first describe foundation definitions and then we will provide a formal definition of FS-NER. Contextually, we will discuss relatedness and differences with other NLP tasks.

Based on Definition 1, NER can be identified as the task \(T\) and \(E\) is a corpus of sentences annotated with entity mentions, so that a performance measure \(P\) (usually \(Precision\), \(Recall\) and/or \(F1\)) of the machine learning model can be improved. Formally, NER is defined as in Definition 2.

It should be noted that the definition provided above is restricted to continuous spans, which represents a frequent occurrence. However, during the application of NER to real-world tasks, named entities that are overlapping or situated in discontinuous text spans may arise. These scenarios have been extensively investigated in various studies [37, 98, 154].

As a result of its peculiar characteristics, NER is a token-level classification task, where models make their predictions token by token. It may be considered as the parallel task of image segmentation in computer vision, where classification is on a pixel-by-pixel basis rather then relying on the whole image. As a consequence, many approaches and model architectures proposed to deal with few-shot challenges in text classification are not directly applicable to NER, thus requiring further efforts, extensions, and/or adjustments.

Machine learning models usually need for big datasets with supervised information (a.k.a. ground-truth) to be effectively trained. The sub-area of FSL has the aim at obtaining good performance when only a small set of supervised data is available. Formally, it can be defined as in Definition 3.

In concrete terms, FSL aims at learning a classifier able to predict a label \(y\) for each input \(x\) (e.g., image classification [81], text classification [143]). In the light of the definitions discussed above, we can provide a clear and concise definition for FS-NER as described in Definition 4.

The substantial difference in FS-NER is that each input sentence \(\mathbf {s} = [t_1, t_2, \dots , t_N]\) has to be associated to multiple labels \(\mathbf {y} = [y_1, y_2, \dots , y_N]\), one for each token \(t_i \in \mathbf {s}\), by the few-shot model. In addition to the formal task difference, it is important to note that two tokens \(t_i \in \mathbf {s}\) (\(i \in [1,N]\)) and \(t_j \in \mathbf {s}\), (\(j \in [1,N]\)) are not independent but semantically related. As a consequence, NER works usually leverage their relatedness to build effective machine learning models, but this also means that many FSL works may be unsuitable for FS-NER. However, several efforts have been made during this years to adapt and extend FSL methodologies or to borrow key ideas. In this work, we will provide an in-depth analysis of how FSL works have been adapted to FS-NER.

Deep neural networks and Transformer architectures represent the foundation of contemporary NER techniques, which require little to no feature engineering to achieve cutting-edge benchmark performances. However, since they typically require enormous hand-labeled training sets, these gains are frequently challenging to apply in real-world situations. In FS-NER, the number of available training examples is small, thus impacting the reliability of the resulting NER model which, as a consequence, usually overfits data [157].

We empirically show this in Figure 2, where we report trends of F1 scores as the NER training epochs increase on validation sets from three widely used benchmark datasets: CoNLL-2003 [148], WNUT-17 [28], WikiANN (en) [107].³

We can observe that performance has a decreasing trend when the number of shots is too small — \(shots \lt = 20\) on CoNLL-2003 and WikiANN, \(shots \lt = 100\) on WNUT-17 data —, meaning that the model is overfitting on its training set. Moreover, even when the F1 score increases as the training progresses, it reaches low values due to the inability of the resulting model to recall all the ground-truth entity mentions contained in the test set. From these two challenges — i.e., reducing overfitting and improving performance of NER models when only few labeled examples are available — arises the need for FS-NER methodologies.

In broad terms, FS-NER is needed in every application scenario affected by the rareness of training samples, which is usually due to:

Hence, many real-world applications involve FS-NER. However, current state-of-the-art does not offer many successful use cases yet — methods are usually tested on benchmark datasets. Wang et al. [159] experiment their few-shot method on a private corpus of 1,600 de-identified EHRs from cardiology, respiratory, neurology and gastroenterology deparments; Ni et al. [102] test their cross-lingual FS-NER approach on a custom multi-lingual dataset with over 50 entity types annotated to build cognitive question answering applications on top of the FS-NER system.

In this section, we propose a taxonomy to categorize existing state-of-the-art FS-NER techniques, as shown in Figure 3. The first layer of the taxonomy is based on whether the methodology is focused on model architecture (model-centric) or data (data-centric), respectively. Input data sources and methodological flows change dependently on the technique, but we summarized high-level details in Figure 4.

In all the fields of Machine Learning, Transfer Learning is the standard approach to deal with the lack of data. The knowledge — i.e., their learned parameters — of models trained on huge datasets is “adapted” with new training iterations to make it possible for the model to perform well with a target domain where there is a lack of resources. Current Transfer Learning methods for NER are mainly based on deep neural networks and Transformer architectures and usually leverage feature representation transfer [109], which makes the model learn to map inputs from different domains in a close feature space, and parameter transfer [168], which makes the target model parameters close to those of the source model. We divide transfer learning approaches for FS-NER in three categories: cross-domain, cross-lingual, fine-tuning. In the following, we describe them in detail.

Inspired by human intelligence, meta-learning (a.k.a. learning to learn) [128, 147] aims at quickly adapting models to new tasks without the need to re-train them from scratch and with only few data points. As humans, we are able to easily learn new skills after a few minutes or zero experience: for example, if we can ride a bike, we will easily learn to ride a motorcycle. This can be accomplished by ML models with a meta-learning phase during which the model learns to adapt to a large variety of tasks.

Since when deep learning has started to proliferate, model architectures have been the focus of research. In the context of FS-NER, works are based on transfer or meta learning. The two approaches, while similar in their objective (i.e., improving the performance in the presence of scarse data resources), are slightly different in the mode in which they operate: while transfer learning uses a model trained with data from a similar domain or language to shift its knowledge to another model, meta learning focuses on training procedures allowing models to quickly adapt to new tasks.

When a model trained on a similar task is available, transfer learning can be easily applied without too many adaptation, and guarantees high-quality results, especially with transformer architectures. However, this ideal scenario is quite uncommon, since real-world task may be similar but deal with another language, or other entity types, thus requiring many adaptations which may also limit the resulting performance — e.g., multi-lingual transfer usually requires a machine-translation step whose error inevitably propagate across the transfer-learning framework. To overcome this, meta-learning models easily adapt when new tasks emerge (e.g., a new entity type is required to be recognized).

One common way to deal with the lack of data is data augmentation, which consists in increasing the size of the available dataset with new samples generated by means of heuristics or external data sources. Augmentation methods explored in current literature for NLP tasks usually manipulate words in the original sentence by word replacement [13], random deletion [160], word position swap [93], and generative models [171]. Applying these transformations to NER input samples is not possible due to the token-level classification implied by this task (each manipulation impacts labels). Thus, data augmentation techniques for NER are comparatively less studied [25]. In the following, we describe current methods applying data augmentation in few-shot scenarios. We provide an example of an augmented sample for each method in Figure 11. Data Boost [85] explores the text generation ability of Language Models to generate augmented samples. In particular, GPT-2 [113] is used as a conditional generator and guided toward specific class labels by means of a Reinforcement Learning (RL) approach. A RL stage is added in between the softmax and argmax function of the conditional generator. The state at step \(t\) is the generated sentence before \(t\) \(s_t = \mathbf {x}_{\lt t}\), where \(\mathbf {x}_{\lt t} = \lbrace x_0, x_1, \dots , x_{t-1}\rbrace\); the policy \(\pi _\theta\) is the probability that token \(x_t\) is chosen (action \(a_t\)), i.e., the softmax output of the hidden states \(\pi _\theta (a_t|s_t) = softmax(h^\theta _{\lt t})\). Reward, following Proximal Policy Optimization (PPO) [130] for a given conditional token \(x_t^c\) is computed as follows:where \(G(x_t^c)\) is the salience gain which quantifies the generated token resemblance to target label lexicon by a logarithm summation of cosine similarity with each word in the salient lexicon; KL is the Kullback-Leibler (KL) divergence between conditional \(\theta _c\) and unconditional \(\theta\) distributions; \(\beta\) is a weighting parameter. However, this method has been conceived for text data augmentation, and does not directly apply to NER, as it does not provide the mapping of the entity mention from the original to the augmented sentence.

Counterfactual Generator ¹⁴ [176] addresses the poor generalization ability of few-shot systems to spurious correlations between the entities and their contexts. For example, in the sentence “John lives in New York”, the entity “New York” and its context (“John lives in”) are highly correlated, but it is not true that one causes the other. Thus, based on the idea that entities and contexts are not in causal relationships, Zeng et al. [176] provide a framework to generate weakly-labeled counterfactual examples in few steps: (1) a vocabulary with the desired entity type is prepared, (2) an entity is randomly selected from the input sentence and (3) replaced with a different entity from the entity set to form a counterfactual example; finally, (4) a discriminator model is trained to distinguish good examples from counterfactuals: if the discriminator, i.e., a NER model, can correctly recognize the replaced entity, the counterfactual example is considered in the new training set.

Dai et al. [25] investigate the adaptation of many simple data augmentation methods for NER problems. They are listed as follows:

Experimental results show that all the investigated data augmentation techniques are effective in few-shot settings (e.g., 50- and 100-shot scenarios), but they often fail on full datasets due to the degrading effects of noise.

COSINER [7] is a more sophisticated variation of Mention Replacement (MR) [25]. It employs the cosine similarity to replace mentions with those that are the most similar, as opposed to randomly replacing them. Concept embeddings \(V_{concept}\) extracted from each mention in the training set are used to calculate similarity and computed as follows:whereand\(V_{context}\) being the embedding extracted from a transformer network used as a feature extractor. The similarity-based nature of this method shows improvements in performance w.r.t. techniques based on random replacements.

StyleNER ¹⁵ [18] learns patterns (e.g., style, noise, abbreviations) that differentiate data from high-resource and low-resource domains, and a shared space where they are aligned. The key idea is that text data may differ in textual patterns (e.g., long and complex sentences from research abstracts versus social media data), but their semantics are transferable. Given a source domain \(D_{src}\) and a target domain \(D_{tgt}\) dataset, sentences are linearized by inserting entity labels before their corresponding text span, and a random pair of sentences is then extracted from the two datasets and provided as input to the model. The model works in two steps:

A discriminator is also trained to distinguish the domain of an embedded sentence, which allow to determine if the encoder can generate meaningful representations or the model has to bypass the intermediate mapping step between domains.

In few-shot scenarios, labeled data could be retrieved from heuristics, different domains or laguages, external knowledge bases, or ontologies. Distant supervision [94] aims at leveraging such resources to heuristically annotate training data. For example, in biomedicine there are a lot of curated resources available: NCBO Bioportal [161] houses 541 biomedical ontologies, Medical Subject Headings(MeSH)¹⁷ is a controlled vocabulary with 347,692 classes of medical items, and so on. Combining ontologies is a difficult task due to their heterogeneous structures, concept granularities and overlaps or conflicts between definitions of entities. Generally, the main steps of distant supervision are (1) candidate generation, i.e., the identification of potential entities, and (2) labeling heuristics to generate noisy labels, as shown in the example of Figure 12. The use of distant supervision for FS-NER in low-resource languages has yet to be deeply explored. The amount of external information available in low-resource settings might be very limited: for example, the Wikipedia knowledge graph contains 4 million person names in English while only 32 thousand in Yorùbá [1]. Furthermore, without further tuning under better supervision, distantly supervised models have low recall [14]. In the following, we describe methods that apply distant supervision to NER tasks. SwellShark ¹⁸ [38] aims at automatizing the generation of candidates and noisy labels without hand-labeled data. Its inputs are (1) a collection of unlabeled documents and (2) some form of weak supervision, typically ontologies and heuristic rules. The candidate generator \(\Gamma _\Theta\) is defined as the mapping function from a document collection \(D\) into a candidate set \(\Gamma _\Theta : D \rightarrow \lbrace x_1, x_2, \dots , x_N\rbrace\), where each candidate \(x_i\) is a character-level span within the document. Candidates in SwellShark are determined by heuristics, e.g., dictionary of noun phrases matching with regular expression. To filter actual entities from candidates, a labeling function generator \(\Gamma _\lambda\) is defined as a function which receives a resource \(R\) for weak supervision (e.g., ontology, term frequencies) and generates labeling functions \(\Gamma _\lambda : R \rightarrow \lbrace \lambda _1, \lambda _2, \dots , \lambda _N\rbrace\) (e.g., lexicon matching, frequency-based thresholds).

Choosing heuristics intrinsically involves a tradeoff between development time and performance: greedy heuristics such as dictionary matching often imply low recall, while noun phrases candidates generate large sets which usually consist in an increase of noise, and hand-tuned heuristics may indeed result in high performance, but require more efforts.

AutoNER ¹⁹ [134] handles the problems of incompleteness and noise in automatically-labeled NER data, which characterize methods based on the detection of entity spans with heuristic rules, such as regular expressions [38, 120] and exact string matching [43, 51]. Nevertheless, unmatched tokens in are simply ignored when entities are not covered by the used ontology, introducing many false-negative labels (incompleteness). Furthermore, these methods often require expensive expert effort to cover many special cases. To handle these problems, AutoNER marks some high-quality out-of-dictionary phrases as “potential entities” without requiring human effort. In particular, to leverage the information embedded in dictionaries, AutoNER proposes the tie or break tagging scheme, which tags two adjacent tokens as (1) tie if they belong to the same entity type, (2) unknown if at least one of them belongs to an unknown-type phrase, and (3) break otherwise. This scheme is used for entity span detection, while entity types are then identified based on feature vectors of candidates. Predicting whether two adjacent tokens refer to the same entity or not, AutoNER builds more robust distant supervised models (since it is often noisy on boundaries of entity mentions but not in their inner ties).

One of the main advantages of AutoNER is that entity mentions may be marked as unknown, allowing us to include token which we are unable to identify their types based on distant supervision. For example, “prostaglandin synthesis” may be present in both disease and chemical lexicons, or lexicon may not cover all the possible entity types. AutoPhrase [133] is used to automatically mine new phrases from unlabeled text and a dictionary of high-quality phrases. All the out-of-dictionary phrases are then labeled as unknown and added to the dictionary.

Yang et al. ²⁰ [167] also addresses the incomplete and noisy annotation problems. Weak sentences are obtained by using a dictionary \(D\) built from named entities available in the training dataset \(H\) to weakly annotate a large unlabeled pool of sentences. Note that in this work “distant” resources are used to augment the training set, but the dictionary is built by relying on the available data only, which may be a limit in highly-constrained few-shot settings, where training data do not cover all the possible named entities. To handle the incomplete annotation problem, sentences are allowed to have partial annotations: some token spans are annotated with definite labels, while all the others are associated to all the possible labels. The noisy annotations problem is handled with a reinforcement learning approach which follows Feng et al. [34] to obtain clean instances from distantly supervised NER data. In particular, the state \(\mathbf {S_t}\) is a vector containing: (1) representation of the current sentence obtained with a BiLSTM layer, (2) label scores computed with a MLP layer from the shared encoder, and (3) distant annotation of the instance. The action \(a_t \in \lbrace 0,1\rbrace\) indicates whether to select the \(t\)th distantly supervised sentence, and the policy is learned by optimizing NER performance (reward).

Knowledge-Augmented Language Model (KALM) [80] augments a traditional language model with a knowledge base without requiring any additional component; in addition, it learns to recognize entities in an entirely unsupervised way by using entity type information which is latent in the model. In particular, KALM has the ability to predict masked words from a vocabulary \(V_g\) like any other language model, but it has a separate vocabulary \(V_i\) for each entity type and is able to predict whether to expect an entity from the context. Formally, given a latent variable \(\tau _i\) denoting the entity type \(i\) and previously observed words \(c_t = [y_1, y_2, \dots , y_{t-1}, y_t]\), the probability of the next word is computed as follows:where \(P(y_{t+1} | \tau _{t+1}, c_t)\) is the distribution of entity words of type \(\tau _{t+1}\) and \(P(\tau _{t+1}=j | c_t)\) is the probability that the next word has a given type \(j\). Both are computed as in standard language models, i.e., by projecting the hidden state of a LSTM model and normalizing with a softmax. However, the base model is enhanced by using as input not only the embedding vector of an input word, but also the embedding of the type of the previous word: in this way, KALM is able to model context by taking count of entity types, and it allows to learn latent types more accurately during training.

Cao et al. ²¹ [14] try to maximize the potential of automatically weakly labeled data (e.g., anchors from Wikipedia) by dealing with the incomplete and noisy annotations problems. To obtain a high recall, the framework generates as many weakly-labeled data as possible with a label induction approach assigning labels to words based on Wikipedia anchors and taxonomy, and a data selection scheme which computes a scoring function to distinguish high-quality data from weak sentences and a neural model is then trained on such data. For data selection, the same approach as Ni et al. [102], which is based on annotation confidence and coverage, is applied. For sequence labeling from high-quality weakly-labeled data, Partial CRFs [146] are employed, while a classification module regards name tagging from noisy weakly-labeled data as a multi-label classification problem predicting each word label separately. The first layers of the two network are shared so as to allow knowledge distillation.

Graph Attention Model ²² [87] uses a domain-specific 2dictionary to create a word formation graph which captures variants of entities and thus discovers as new entity mentions as possible. In particular, the vertex set contains all words in the dictionary, which are connected by undirected edges if they appear in the same entity type. Mention candidates are then extracted with a graph-matching algorithm. After extracting all the candidates, a word-mention graph integrates word formation of candidate entities into their sentences: the vertex set contains words in the input sentence and mention candidates extracted. In this way, links between words allows us to capture contextual information, while word-mention links capture the semantic of mentions. Graph information is then leveraged by a learning model including (1) a word embedding layer to represent words, (2) a BiLSTM layer to capture contextual information, and (3) a graph attention network (GAT) [151] to incorporate the information of mention candidates.

Linked HMM ²³ [123] claims that the standard approach of generating candidate spans and then independently labeling each candidate, as in SwellShark [38], limits the applicability to tasks where candidate generators exist, and increases human efforts. Furthermore, candidate generators should identify all the possible entity spans, since errors propagate in the pipeline. Hence, Linked HMM allows users to write multiple rules which provide partial tags to sequences, whose accuracy is estimated by using an identifiable probabilistic generative model without labeled training data. The estimated posterior distribution is used over the true tags to train a sequence tagger. The rules which can be provided are categorized into two types: (1) tagging rules, which vote on the correct tags of sequence elements (they take a training input sequence and output a sequence of the same length indicating their votes on the true tags); (2) linking rules, which vote on whether an adjacent element should have the same or different tag (they allow distant supervision to propagate along sequences in an user-controlled way). These two types of rule are then used by a linked Hidden Markov Model to estimate the true tags for training by reconciling incomplete and conflicting information from multiple rules provided by users.

Consensus Network ²⁴ [67] can be trained on imperfect annotations from multiple sources (e.g., crowd annotations, cross-domain data) by learning representations for each source and dynamically aggregating them by a context-aware attention mechanism. This is based on the intuition that different sources of supervision may have different strengths based on scenarios where they are applied. The framework first uses a multi-task learning schema based on a BiLSTM-CRF network to decouple model parameters into a set of annotator-invariant model parameters and a set of annotator-specific representations, then trains a context-aware attention module for a consensus-based representation by combining predictions on the target data. Specifically, scores from the annotator \(k\) are obtained by combining emission and transformation score matrices from the BiLSTM-CRF network with annotator-dependent matrix \(\mathbf {A}^{(k)}\) which represents the pattern of annotation bias, i.e., the entry \(\mathbf {A}^{(k)}_{ij}\) is the probability of assigning the wrong label \(j\) instead of the correct label \(i\). Scores from different annotators are then combined with weighted voting, where the weight is given by F1 scores on the training set, and an attention module is added to improve generalization, since it provides more weight to sources which are more related to the input sentence.

Active Learning aims at selecting informative sets of examples for training, actively querying the user for labels, as shown in the cycle depicted in Figure 13. The most common approach is uncertainty sampling [72], in which the model selects examples based on the uncertainty of its predictions (a general approach uses entropy as an uncertainty measure [136]). While its theoretical properties have been extensively studied in past works [4, 6, 9, 27], Active Learning approaches are recently spreading in natural language processing tasks. Zhang et al. [180] are the firsts to investigate active learning for sentence classification: they use the Expected Gradient Length(EGL) [132] as an active learning strategy aiming to select the instances which would result in the maximum change in the current model parameter estimates if their labels were provided. In the following, we will describe current applications of Active Learning to NER tasks.

Shen et al. [135] are the firsts to explore AL methods on Deep Neural Networks (DNNs) for the NER task. Due to the complexity of DNNs, the model is not retrained in each active learning round, but it is incrementally trained with each batch of new labels. The work uses uncertainty-based sampling strategies by considering three ranking methods: Least Confidence (LC) [23], Maximum Normalized Log-Probability (MNLP), and Bayesian Active Learning by Disagreement (BALD) [41]. Specifically, LC uses the probability of not predicting the most confident sequence from the model to sort samples in descending order:where \(y_1, \dots , y_n\) is the sequence of slot labels and \(\mathbf {x} = \lbrace x_i\rbrace\) is the sentence, i.e., a sequence of tokens \(x_i\), \(i \in \lbrace 1, \dots , n\rbrace\). Since this approach naturally favours long sentences (which is a downside since they require more effort to be annotated), MNLP is proposed:

Finally, BALD uses a set of M models \(P^1, P^2, \dots , P^M\) sampled from the posterior. Uncertainty is then computed as the fraction of models which disagreed on the most popular choice:where \(|\cdot |\) is the cardinality of a set.

Monte Carlo dropout [40] is used to sample from the posterior. Results show that the performance of the best model trained in a standard supervised fashion is almost reached with approximately 30% of training data.

Self-training, also referred to as self-learning, is an approach similar to distant supervision, where the model is trained on examples labeled by the model itself. As shown in Figure 14, the difference is that the labeling heuristics is replaced by the model itself. Originally proposed by Scudder et al. [131], it is one of the earliest semi-supervised methods. In the NLP field, it has been successfully applied to neural machine translation [50] and sentence classification [99]. Zoph et al. [183] show that self-training guarantees improvements in performance in both high- and low-data scenarios, while data augmentation results sometimes in decreases of performance of pretraining. In the following, we explore self-training approaches for FS-NER. LM-LSTM-CRF ²⁵ [84] leverages the knowledge obtained with a language model trained in an unsupervised fashion to improve sequence-labeling performance. It leverages word-level and character-level information of input-samples in a co-training fashion, i.e., each input is represented by different sets of features, each of them providing complementary information. Both a language model and a sequence-labeling model share the same character-level layer, which consists in two LSTM units learning character-level information in a completely unsupervised way to capture style and structure of input texts. However, since the two task handled are not strongly related, empirical results show that this can hurt the overall performance. Hence, outputs of character-level layers are transformed into task-specific spaces, so that the language model can indirectly provide its knowledge to the NER model thanks to the shared layer, without sharing its feature space.

Chen et al. [15], considering that performance of self-training highly depends on how new data are selected, use a reinforcement learning to learn how to select instances from an unlabeled pool to be added to the training set in self-training scenarios. The NER model, which had been previously trained with few samples, will classify the new selected samples and be retrained. The self-training approach is considered as a decision process, described as a function that receives the self-labeled instances as inputs and outputs the acceptance or rejection of such instances to the training set. A Deep Q-Network (DQN) [96] then learns the selection strategy based on performance improvements on a development set. The Q-function is implemented using a neural network with three layers which receives the state \(s = (h_s, h_c, h_p, h_t)\) of the learning framework as input, where \(h_s\) is a representation of the input sample, \(h_c\) is the confidence of tagging the instance using the model, \(h_p\) is the marginals of the prediction, \(h_t\) is the hidden representation from the model. The Q-function \(Q^\pi (s,a)\rightarrow R\) receives the state \(s\) and an action \(a\) as inputs and returns a reward as a result of the execution of \(a\). The policy \(\pi\) has the aim at maximizing the reward of actions. In this work, the reward is defined as the difference in NER performance when adding a new sample (i.e., the action) to the training dataset. This is a similar setting to Fang et al. [33], who apply DQNs to find the best Active Learning policy.

RDANER ²⁶ [174] uses a bootstrapping approach to obtain model predictions on easily-obtainable unlabeled data and retrain the model with the augmented weak dataset. Firstly, a general domain pre-trained language model \(\mathcal {M}\) is fine-tuned on the few-shot labeled corpus available \(\mathcal {D}^L\). Then, \(\mathcal {M}\) is used to annotate an unsupervised dataset \(\mathcal {D}^U\) to obtain a weakly annotated dataset \(\mathcal {D}_{weak}^U\) which is then combined with \(\mathcal {D}^L\) to get the augmented corpus. A threshold \(\theta\) is used to filter out tags with low probabilities assigned by \(\mathcal {M}\). This process is iterated until the achievement of an acceptable level of accuracy or the maximum number of iterations. Experiments show that this simple approach has worse F1-scores than domain-specific pre-trained language models (e.g., BioBERT [69], SciBERT [8]), but it could be a good alternative in low-resource languages and domains where huge domain-specific language model variants are not available.

BOND ²⁷ [79] combines a self-training approach with distant supervision. During the first stage, distant labels are generated with external knowledge bases and a pre-trained BERT model is adapted to the distantly supervised NER task with early stopping. In the second stage, a teacher-student framework is employed, where the student model \(\theta ^{stu}\) is trained with pseudo-labels generated by the teacher model \(\theta ^{tea}\). The teacher is initialized with weights \(\hat{\theta }\) learned during the first stage, while the student may be initialized with \(\hat{\theta }\) or pre-trained BERT layers \(\theta ^{BERT}\). At the \(t\)-th iteration, the teacher generates weak labels which the student learns to fit. Then, teacher and student are updated \(\theta ^{tea}_{(t+1)} = \theta ^{stu}_{(t+1)} = \hat{\theta }^{tea}_{(t)}\). In this way, pseudo-labels are progressively refined so that the student can exploit their knowledge and avoid overfitting.

Huang et al. ²⁸ [57] leverage the labeled training set \(\mathcal {D}^L\) and all the available in-domain unlabeled samples \(\mathcal {D}^U\) by resorting to the knowledge-distillation approach proposed by Xie et al. [164] in image classification. The algorithm operates in three steps: (1) a teacher model \(\theta ^{tea}\) is learned with labeled tokens \(\mathcal {D}^L\); (2) the teacher model is used to generate soft labels \(\mathbf {\hat{y}_i}\) on unlabeled tokens \(\mathbf {x}_i \in \mathcal {D}^U\), \(\mathbf {\hat{y}_i} = f_{\theta ^{tea}}(\mathbf {x}_i)\); (3) a student model \(\theta ^{stu}\) is trained on labeled and unlabeled tokens with a composed Kullback-Leibler loss \(\mathcal {L}\):where \(\lambda ^U\) is a weighting hyper-parameter. Experimental results show that self-training usually results in significant performance improvements also when combined with noisy supervision.

Data-centric approaches for FS-NER try to make the most of the few available training samples or to leverage in the cleanest way possible an available but unannotated corpus, thanks to the intervention of humans, external resources, and/or the model itself.

When the available few-shot training corpus is the one and only source of information available, data augmentation can be usually applied since it increases the size of the dataset by transforming the available samples. However, the majority of research work assumes the presence of external resources such as a vocabulary of entities [176] or synonyms [25].

In general, an unannotated corpus is the key to achieve better results in few-shot contexts — methods basically differ on how they handle it to find greedy annotations. Active learning usually requires one (or many) human (s) to optimize the tradeoff between the annotation efforts required and the resulting performance. On the other hand, distant supervision leverages external resources, such as heuristic rules and ontologies, to obtain weak labels. Similarly, self-learning gets weak labels by the model itself, which could be particularly useful when using language models to leverage the knowledge they acquired during the pre-training stage.

As we can see in Figure 2, the number of training examples needed for few-shot learning to attain good performance might vary across different datasets and tasks. The complexity of the task, the diversity of the instances in the training set, and the similarity between the training and test sets are only a few of the factors that may contribute to this variability.

To learn effectively, tasks that are more complicated or call for a higher degree of abstraction would need more training instances. Several examples could be required to learn the pertinent aspects, for instance, if the task entails identifying complicated patterns or mastering sophisticated decision rules.

The model may be able to acquire more generalizable features and need fewer examples to perform well if the training set includes a broad group of examples that span a wide range of variations. The model could need more examples if the training set is wider or more homogeneous so that it can learn enough variation.

Additionally, the similarity between the training and test sets can also play a role. If the test set is very different from the training set, the model may need more examples to generalize to new examples in the test set. On the other hand, if the test set is similar to the training set, the model may be able to achieve good performance with fewer examples.

Understanding the factors that affect the number of training examples required for FS-NER learning can help researchers develop new approaches to improve model performance in low-data regimes. While one potential area of research could focus on developing methods that can learn more complex and abstract representations from few examples, we argue that another possible direction could involve exploring ways to generate more diverse training sets, such as using generative models to augment existing examples or incorporating external knowledge sources to provide additional training data. While generative models and distant supervision have been already investigated in FS-NER, their ability in generating heterogeneous training sets has yet to be explored.

Despite the growing number of research studies on FS-NER, comparing their performance is a challenge due to the lack of a unified evaluation protocol. With regards to the datasets used to experiment methods, we have identified a set of benchmark corpora used in most of the works: CoNLL-2003 [148], SNIPS [21], Ontonotes [111], WikiANN [107].

However, research in this field has not reached a consensus yet on the three aspects described below:

Although there is a growing body of research on FS-NER, the applicability of the proposed techniques in real-world scenarios where the rareness of training samples is involved has never been studied in-depth. Current works experimenting their techniques with real use cases usually focus on healthcare: Wang et al. [159] use Chinese EHRs data from an affiliated hospital with 1600 de-identified clinical notes from departments of cardiology, respiratory, neurology and gastroenterology; La Gatta et al. [42] experiment with an Italian dataset of cardiological clinical notes. However, we argue that the need for FS-NER methodology is more widespread and future works should also focus to other domain (e.g., finance). To the best of our knowledge, the only other work using FS-NER approaches to solve a real-world problem is from Ni et al. [102] who create a custom multi-lingual dataset (i.e., Japanese, Korean, German, Portuguese) with over 50 entity types to build cognitive question-answering applications.

The progress of Natural Language Processing (NLP) has been propelled by the creation of generative Large Language Models (LLMs), which can be seen in studies such as Devlin et al. [30], Brown et al. [11], and Chowdhery et al. [19]. LLMs have already become a crucial component of many commonly used products, such as the coding assistant Copilot [17], the Google search engine,²⁹ and the more recent addition of ChatGPT.³⁰ However, it is not immediately clear how these models may impact future research works in few-shot Named Entity Recognition (NER).

One potential way that generative models could impact FS-NER is by generating synthetic training data. Synthetic data can be used to augment limited training datasets, which is particularly useful in few-shot learning scenarios. By using generative models to generate synthetic examples, it may be possible to improve the ability of FS-NER models to generalize to new, unseen entities. StyleNER [18] (described in Section 4.1), is a first attempt to use generative models for FS-NER, but it relies on external data to learn their patterns and generate new samples that imitate their style. This poses many challenges on the choice of the external data to be used and their effectiveness on the few-shot problem at hand.

Another way that generative models could impact FS-NER is through their ability to generate context-aware representations. Named entities are often highly dependent on the context in which they appear, and the ability to generate context-aware representations may be useful for FS-NER models. By leveraging the power of generative models, it may be possible to learn more nuanced representations of named entities and their associated contexts.

Overall, there is still much work to be done to fully investigate the possible effects of generative models on FS-NER. Yet, it is evident that these models have a lot of potential for future advancements in FS-NER and other natural language processing tasks. The optimal techniques to utilize generative models in the context of few-shot learning will need to be determined through thorough experimentation and review, as with any new technology.

While standard procedures to train deep-learning models have shown robust performance improvements over time, there is an increasing research interest toward developmental approaches [64, 106, 165, 182, 184]. Smith et al. [137] show that two-year old children can infer a new category from only one instance. This is presumably due to the fact that the human brain, during early learning, is trained to develop foundational structures to learn the learning procedure [68], which is similar to the key-idea behind meta-learning. Instead of learning from scratch, neural networks should be able to recognize regularities from past tasks to help them solving new tasks [64]. Due to these parallelisms, current research in developmental psychology could guide future works in FS-NER. For example, Orhan et al. [106] and Xing et al. [165] leverage experiments showing that learning object names can change the visual features children use for word learning, thus recognizing that language helps recognizing new visual objects.