A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT

Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks across different data modalities. A PFM (e.g., BERT, ChatGPT, GPT-4) is trained on large-scale data, providing a solid parameter initialization for a wide range of downstream applications. In contrast to earlier methods that use convolution and recurrent modules for feature extraction, BERT learns bidirectional encoder representations from Transformers, trained on large datasets as contextual language models. Similarly, the Generative Pretrained Transformer (GPT) method employs Transformers as feature extractors and is trained on large datasets using an autoregressive paradigm. Recently, ChatGPT has demonstrated significant success in large language models, utilizing autoregressive language models with zero-shot or few-shot prompting. The remarkable success of PFMs has driven significant breakthroughs in AI, leading to numerous studies proposing various methods, datasets, and evaluation metrics, which increases the demand for an updated survey. This study provides a comprehensive review of recent research advancements, challenges, and opportunities for PFMs in text, image, graph, and other data modalities. It covers the basic components and existing pretraining methods used in natural language processing, computer vision, and graph learning, while also exploring advanced PFMs for different data modalities and unified PFMs that address data quality and quantity. Additionally, the review discusses key aspects such as model efficiency, security, and privacy, and provides insights into future research directions and challenges in PFMs. Overall, this survey aims to shed light on the research of the PFMs on scalability, security, logical reasoning ability, cross-domain learning ability, and user-friendly interactive ability for artificial general intelligence.

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

1. https://openai.com/blog/chatgpt/.

2. March 2023.

3. https://blog.google/technology/ai/bard-google-ai-search-updates/.

1. Ce Zhou, Qian Li, Chen Li, Jun Yu and Yixin Liu contributed equally to this work.

Authors and Affiliations

1. Michigan State University, East Lansing, MI, USA

   Ce Zhou, Guangjing Wang & Qiben Yan

2. Beihang University, Beijing, China

   Qian Li, Chen Li, Cheng Ji, Hao Peng & Jianxin Li

3. Lehigh University, Bethlehem, PA, USA

   Jun Yu, Yixin Liu, Kai Zhang, Lifang He & Lichao Sun

4. Macquarie University, Sydney, Australia

   Jia Wu

5. Nanyang Technological University, Singapore, Singapore

   Ziwei Liu

6. University of California San Diego, San Diego, CA, USA

   Pengtao Xie

7. Salesforce AI Research, Palo Alto, CA, USA

   Caiming Xiong

8. Duke University, Durham, NC, USA

   Jian Pei

9. University of Illinois at Chicago, Chicago, IL, USA

   Philip S. Yu

Corresponding authors

Correspondence to Ce Zhou or Qian Li.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A Basic components

1.1 A.1 Basic components on NLP

1.1.1 A.1.1 Language model

With the rapid development of deep learning, LMs are more and more applicable to the pretraining of NLP models. The LM can estimate the probability of rationality of a paragraph of the text. There are two main types of LMs: statistical LM and neural network LM.

Statistical LM The statistical LM is a mathematical model to solve the context-related characteristics of natural language from the perspective of probability and statistics. The core of statistical LMs is to determine the probability of a sentence appearing in a text. As the theoretical basis of the probabilistic LM, the N-gram model profoundly influences the subsequent LM. It plays a pivotal role in the field of the LM. The N-gram LM introduces the Markov hypothesis, which assumes that the probability of the occurrence of the current word only depends on the nearest \(n-1\) words. The maximum likelihood probability of a word \(w_{i}\) can be calculated by

where \(T=[w_{1}, w_{2}, \ldots , w_{N}]\) is the text sequence and \(C(w_{i-n+1}, w_{i-n+2}, \ldots , w_{i})\) is the co-occurrence frequency of \((w_{i-n+1}, w_{i-n+2}, \ldots , w_{i})\). The \(p\left( w_{i} \mid w_{1}, w_{2}, \ldots , w_{N} \right)\) is calculated according to the chain rule

N-gram uses the probabilities of each word in the sequence to represent the co-occurrence probability of the whole text sequence. When N is large, it indicates a more vital constraint on the occurrence of the next word in the sequence and leads to more sparse frequency information. When N is small, the statistical results have higher reliability and better generalization ability, but the constraint will be weaker.

Neural LM The statistical LM adopts maximum likelihood estimation, which is intuitive and easy to understand. However, there are still problems such as a lack of long-term dependence, the rapid growth of parameter space and sparse data. Therefore, the neural network is introduced to map the LM to a continuous space. Neural LMs use distributed representations of words to model natural language sequences. Unlike class-based N-gram models, neurolinguistic models are able to recognize two similar words without losing the ability to encode each word as different from the other. It can be directly used for NLP tasks. It mainly introduces Forward Feedback Neural Networks (FFNN), Recurrent Neural Networks (RNN), and pretrained LMs.

The model architectures of forward feedback neural network, recurrent neural network and Pretrained LMs. \(H^{1,2}\), \(H^{2,3}\) and \(H^{1,3}\) are the weight matrices used to connect each layer

Full size image

As shown in Fig. 15a, FFNN according to the all former words of \(x=[w_{1}, \ldots , w_{i-1}]\) calculates the probability of \(w_{i}\). In order to predict the conditional probability of \(w_{i}\), x is sharing the projection matrix \(M \in R^{|V| \times m}\) to a continuous feature vector space according to the projection index, |V| is word library size, m is the dimension of the feature vector. The output is represented as

where \(H^{1,2}\), \(H^{2,3}\) and \(H^{1,3}\) are the weight matrices used to connect each layer, and \(b_{1}\) and \(b_{2}\) are the bias values of the hidden layer and the output layer respectively.

The structure of the FFNN contains only limited information about the foregoing and has some limitations on the length of the input sequence. Therefore, the RNN LM comes into being. As shown in Fig. 15b, RNN can accept input of any variable length. When the input window is moved, its internal state mechanism can avoid repeated calculation, and parameter sharing further reduces the number of model parameters. Therefore, compared with FFNN, RNN has a great advantage.

The pretraining LM is to get a set of model parameters by pretraining some tasks. It initializes the model with these parameters and then trains to improve the model performance effectively. The commonly used pretraining models are fixed embedding (Word2vec [12], Glove [69], etc), variable embedding (Embeddings from LMs (ELMO) [303], Generative pretrained Transformer (GPT) [50] and Bidirectional Encoder Representations from Transformers (BERT) [13], etc). Here, we give an example of the GPT model, as shown in Fig. 15c. It adopts a two-stage process. In the first stage, the Transformer decoder is used as the basic unit of the model to perform text prediction. In the second stage, the GPT is initialized differently for different downstream tasks, training the model and fine-tuning the parameters.

1.2 A.2 Basic components on GL

Due to the extensive use of graph data in many fields, some communities (e.g., chemistry, protein, and social network) have recently focused on the study of graph pretraining. These pretraining models encode graph attributes, structures, and other information into node representations from multiple perspectives by designing different pretext tasks, which are used to optimize downstream tasks. In this section, we introduce the definition of the basic concepts of graphs, and then provide a formal definition of the PFM on the graph.

1.2.1 A.2.1 Notations and definitions of graphs

Unless particularly specified, the notations used in this article are illustrated in Table 4. We use \(\mathcal {G}=\{G_{i}\}^{N}_{i}\) to represent a set of graphs, where N represents the number of graphs. Depending to the graph’s definition of the edges and nodes, graph data can be classified into the following types.

Definition 1

An unattributed graph is \(G=(V,E)\), where \(v \in V\) is a node, \(e \in E\) is an edge, and naturally \(E \subseteq V \times V\). Adjacency matrix \(A \in \mathbb {R}^{n \times n}\) represents the topology of graph G, where \(n=|V|\). \(A_{i,j}=1\) denotes there is an edge between node \(v_{i}\) and \(v_{j}\), otherwise \(A_{i,j}=0\).

Definition 2

An attributed graph is \(G=(V,E,X_{v},X_{e})\), where \(X_{v} \in \mathbb {R}^{n \times d_{v}}\) and \(X_{e} \in \mathbb {R}^{m \times d_{e}}\) are the feature matrices of nodes and edges, \(|V|=n\), \(|E|=m\), \(d_{v}\) and \(d_{e}\) denotes the feature dimensions of node and edge. In fact, in most application scenarios, only nodes have attributes, and edges have no attributes or only weights.

Definition 3

An undirected graph is \(G=(V,E)\), where \(e_{i,j} \in E\) means an unordered node pair \((v_{i}, v_{j})\). In particular, the adjacency matrix A of the undirected graph is a symmetric matrix (i.e., \(A_{i,j}=A_{j,i}\)).

Definition 4

A directed graph is \(G=(V,E)\), where \(e_{i,j} \in E\) means an ordered node pair \((v_{i}, v_{j})\).

Definition 5

G has a node-type mapping function \(f_{v}: V \rightarrow \mathcal {T}^{v}\) and an edge-type mapping function \(f_{e}: E \rightarrow \mathcal {T}^{e}\). When \(|\mathcal {T}^{v}|=|\mathcal {T}^{e}|=1\), the graph \(G=(V,E)\) is a homogeneous graph. In other words, all nodes in G belong to a type, and all edges also belong to one type.

Definition 6

When \(|\mathcal {T}^{v}|>1\) and/or \(|\mathcal {T}^{e}|>1\), the graph \(G=(V,E)\) is a heterogeneous graph. In particular, a heterogeneous graph must be an attributed graph.

1.2.2 A.2.2 Learning settings on graphs

GL methods are usually used to solve machine learning tasks on graph data, and we introduce different settings (supervision mode and learning mode) for GL.

Before that, we first provide the notations of the corresponding mathematical formulation of GL. \(C=\{c_{1}, c_{2}, \cdots , c_{K}\}\) is a set of target components defined in a graph set \(\mathcal {G}\) (\(G^{c_{i}} \in \mathcal {G}\)), and \(c_{i}\) is associated with a corresponding ground truth label \(y_{i} \in \mathcal {Y}=\{1,2,\cdots ,N_{y}\}\), where K denotes the total number of target components, and \(N_{y}\) is the number of classes being predicted. Then the graph data can be represented as \(D=\{c_{i}, G^{c_{i}}, y_{i}\}^{K}_{i}\), and a complete GL model \(M_{GL}\) can also be determined by \(y_{i} = M_{GL}(c_{i}, G^{c_{i}})\). For instance, in a node classification task, \(c_{i}\) is the node to be classified, \(y_{i}\) denotes \(c_{i}\)’s label in graph \(G^{c_{i}}\). Similarly, in a node clustering task, \(c_{i}\) is the node to be clustered, \(y_{i}\) denotes the corresponding cluster label in graph \(G^{c_{i}}\).

Supervision mode

Schematic of different supervision modes

Full size image

Depending on the source and scale of the training data, the supervision settings of GL can be divided into four types as shown in Fig. 16. Supervised GL is the most common mode in the real scenario. Given the target component \(c_{i}\) and the corresponding ground truth label \(y_{i}\), the goal is to minimize the loss function between the predicted label of the GL model (i.e., \(y^{pred}_{i}=M_{GL}(c_{i}, G^{c_{i}})\)) and the expected label \(y_{i}\) of all \(c_{i}\). Compared with supervised learning, unsupervised GL refers to situations in which no labeled data is provided, only the attributes and structure distribution of graph data (i.e., \((c_{i}, G^{c_{i}})\)) can be used. Self-supervised GL is a special case of both supervised and unsupervised learning. Specifically, self-supervised learning mainly uses pretext tasks (e.g., clustering, completion, and partition) to mine its own supervised information (i.e., pseudo-labels) from large-scale unsupervised graph data, and trains the GL model \(M_{GL}\) through the self-supervised information, so that it can learn to the valuable features of downstream tasks. In other words, the supervised information of self-supervised learning is not manually labeled, but the pretext tasks automatically construct supervised information from large-scale unsupervised data for supervised learning or training. Semi-supervised learning is a combination of unsupervised and supervised learning, which aims at learning data distribution to predict unlabeled data to solve the problem of difficulty in obtaining labeled data in real scenarios. In GL, semi-supervised learning refers to the realization of pattern recognition given a few labeled data and mass unlabeled data.

Learning mode The GL model \(M_{GL}\) is optimized by the given training samples, and adjusted on the validation samples to participate in the test. According to the visibility of the graph data at different stages, the learning settings of GL model \(M_{GL}\) can be classified into two categories: inductive learning and transductive learning.

Definition 7

Inductive Learning, which is the most common setting in machine learning tasks, trains the model on labeled data and then tests on samples that have never appeared in the training stage. Formally, given a training sample \(\{(c_{i}, G^{c_{i}}, y_{i})\}^{N_{l}}_{i=1}\), \(\{(c_{j}, G^{c_{j}})\}^{N_{u}}_{j=1}\), where \(N_{l}\) and \(N_{u}\) are the numbers of labeled/unlabeled samples. Inductive learning learns a function \(f^{ind}: \mathcal {G} \mapsto \mathcal {Y}\) so that \(f^{ind}\) is expected to be a good classifier on the future graph data \(\{(c_{k}, G^{c_{k}})\}\), beyond \(\{(c_{j}, G^{c_{j}})\}^{N_{u}}_{j=1}\).

Definition 8

Transductive Learning is different from inductive learning in that all samples are visible during both the training and testing stages. Formally, given a training sample \(\{(c_{i}, G^{c_{i}}, y_{i})\}^{N_{l}}_{i=1}\), \(\{(c_{j}, G^{c_{j}})\}^{N_{u}}_{j=1}\), transductive learning learns a function \(f^{trans}: \mathcal {G}^{l+u} \mapsto \mathcal {Y}^{l+u}\) so that \(f^{trans}\) is expected to be a good classifier on the unlabeled data \(\{(c_{j}, G^{c_{j}})\}^{N_{u}}_{j=1}\).

Under the supervised setting (including semi-/self-supervised), the unified classifier optimization methods of inductive learning and transductive learning can be written as:

where \(\mathcal {L}\) is the cross-entropy loss, \(c_{i}\) can be node, edge or subgraph of its associated graph \(G^{c_{i}}\), and \(f^{(\cdot )}_\theta\) denotes inductive/transductive function with parameter \(\theta\).

Compared with using only one pretext task, some methods have designed some integration mechanisms to incorporate the advantages of multiple pretext tasks into a unified framework.

B Traditional learning methods

1.1 B.1 Traditional text learning

NLP is a research field that integrates linguistics and computer science. Its main research tasks include part-of-speech tagging, named entity recognition, semantic role labeling, machine translation, question answering, sentiment analysis, text summarization, text classification, relationship extraction, event extraction, etc. The LM can be considered the cornerstone of the downstream NLP tasks. It experiences four processes: grammar rule LM, probabilistic LM, neural network LM, and pretraining LM. A PFM trains on a large benchmark dataset to obtain a model that can solve new similar tasks, which has become a new hotspot in current LM research.

Word representations play a significant role in downstream tasks, which is the basis of NLP. The N-gram model preprocesses text features and encodes adjacent N words as a group, which makes it overly dependent on the richness of the training corpus. Otherwise, data-sparse is likely to occur, and the computational complexity will increase exponentially with the increase of N. Neural Network LM (NNLM) [11] adopts the idea of word vector for the first time, and the low-dimensional word vector of distributed representation can solve the discrete problem caused by word embedding well. However, it is still challenging to solve the problem of high computational complexity. The computational complexity of the word2vec model is independent of the selected window size but is determined by the dictionary size and the word vector dimension. Many downstream tasks can be significantly improved by training on a large corpus using word vector embedding after initial training. However, the problem of polysemy for the static word vector is still unsolved, and it still belongs to the shallow LM [304, 305]. Therefore, more effective models are urgently needed to deal with the data set more flexibly. To capture high-level concepts of context, such as polysemy elimination, syntactic structure, etc. Neelakantan et al. [306] propose to learn multiple embeddings per word type. Zhou et al. [307] integrate the features on both dimensions of the matrix to enrich semantics by using subword information. Based on the Continuous Bag Of Words (CBOW) [12] in word2vec, Hui et al. [308] fine-tune the generated word vectors for emotion and obtain the word vectors containing both semantic meaning and emotional tendency, which significantly improved the performance in the Weibo sentiment classification task. Liu et al. [309] propose a model of hierarchical translation for machine translation. It uses the neural LM based on RNN as the word vector generation model. Liang et al. [310] propose an approach based on the double-layer self-attention mechanism for machine reading comprehension, and the model is divided into three parts: single document encoder, multi-document encoder, and answer prediction. In the single document encoder, the problem of the context information is represented by the Gated Recurrent Unit (GRU) model. Zhang et al. [311] propose an INDependent RNN (INDRNN) and attention mechanism for user intention classification, using word vectors generated by word2vec as input. The model introduces a word-level attention mechanism to effectively quantify the contribution of domain vocabulary to the intention category.

1.2 B.2 Traditional image learning

There are several types of neural networks in the deep learning era, from the beginning of most famous convolutional neural networks (CNNs) to the subsequent Attention- and Transformer-based networks. A deep neural network refers to an artificial neural network with more hidden layers, and more parameters are used to represent the target model, which leads to the SOTA performance on the benchmark dataset from image to video. Here, we introduce the milestone networks in CV chronologically.

1.2.1 B.2.1 Convolution-based networks

ImageNet [312], as one of the most important databases in computer vision, has aroused many milestone network architectures in image classification, including AlexNet [313], NIN [314], VGG [315], GoogLeNet [316], ResNet [317], DenseNet [318], etc. When it comes to object detection and semantic segmentation, researchers explore R-CNNs [319,320,321,322], FCN [323], SSD [324], YOLOs [325,326,327,328,329], SegNet [330], PSPNet [331], Deeplabs [332,333,334,335], RefineNet [336], etc. on common benchmark datasets, such as PASCAL VOC [337, 338], MS COCO [339], etc.

There are several shared features among these popular convolution-based networks: (1) data augmentation. Deep models require much more data to fit a complicated model, thus the data augmentation technique such as flipping, rotation, cropping, scaling, translation, and even adding noises enlarges the training dataset; (2) convolution. The convolutional kernel is used to extract the features of original image data, which maintains the spatial structure for the adjacent pixels; (3) deep architecture. The deep architecture contains more parameters, which enhance the capability of the model. These common features have contributed to the SOTA performance of convolutional neural networks (CNNs) in computer vision for nearly recent 10 years.

1.2.2 B.2.2 Recurrent neural networks

Different from CNNs targeting 2D-dimensional image applications, recurrent neural networks (RNNs) [340,341,342] try to use recursive cells to process pictures in sequence, i.e., video data. However, the weaknesses of gradient explosion and long-term dependencies restrict further development of this model. To handle these problems embedded inside the RNN-based models, long short-term memory (LSTM) [343] was proposed by Hochreiter and Schmidhuber in 1997. In addition, the improved capability of LSTMs produces popularity and attracts attention both in NLP and CV [344,345,346,347,348].

1.2.3 B.2.3 Generation-based networks

Generative Adversarial Networks (GANs) [349] have provided a paradigm to learn representations for unlabelled data, and spawn many GAN-based approaches on downstream tasks. In image translation, pix2pix software [350] first proposes the conditional adversarial networks as a solution to the image-to-image translation problems, and achieves reasonable results on real-world datasets. Markovian Generative Adversarial Networks (MGANs) [351] is a method to generate texture synthesis, which can be applied to style transfer and video stylization. CycleGAN [352] provides a learning algorithm to translate an original image from the source domain to a target domain without containing pairs of images in datasets for supervised learning. StyleGAN [353] is a style-based generator to serve as an alternative architecture for traditional GANs. Pixel Recurrent Neural Networks (PixelRNN) [354] aims to complete images by modeling full dependencies between the color channels. DiscoGAN [355] is designed to learn relations between different domains.

GANs have also provided a novel direction to study data synthesis because it perfectly simulates the distribution of the original data. Laplacian Pyramid of Adversarial Networks (LAPGAN) [356] uses a cascade of convolutional networks to generate images in a coarse-to-fine fashion. Similarly, Stacked Generative Adversarial Networks (SGAN) [357] decompose variations into multiple levels and gradually resolve uncertainties by stacking several GANs in a top-down way.

1.2.4 B.2.4 Attention-based networks

Based on the success of CNNs in the area of CV, the attention module is designed to be equipped with the popular CNNs. For example, SENet [358] proposes a channel attention module, which won first place in the competition of ILSVRC2017. In addition, CBAM [359] sequentially infers attention maps along both channel and spatial dimensions. Many innovative works, such as GCNet [360] and CCNet [361], are inspired by this idea of soft-attention mechanism, which outperforms the traditional CNNs on major benchmarks for both recognition and segmentation tasks. In particular, the self-attention mechanism [362], calculating the response at a position among all entities in a sequence by attending to all positions within the same sequence, is proposed to estimate the relevance of one position to other positions in feature maps. To control the expected entities and model more complex relations among different elements in the sequence, masked self-attention and multi-head attention [38] are the key components proposed to substitute the function of convolutions in the era of transformers.

1.2.5 B.2.5 Transformer-based networks

Recently, inspired by the self-attention mechanism and subsequent success of the transformer in NLP, researchers in CV also have tried to use the transformer as an alternative to the convolution. Self-attention-based transformer models always operate in a two-stage training mechanism: (1) pretraining on a primitive dataset (always big but not well labeled) by defining pretext tasks; (2) transferring the pretrained weights to the downstream tasks and adjusting the parameters on the target domain dataset by finetuning. Vision Transformer (ViT) [40] is applied on CV and achieves the SOTA performance on major benchmark datasets. Data-efficient image Transformers (DeiT) [363]was proposed by Facebook AI to train image transformers more efficiently and maintain the SOTA performance simultaneously. DEtection TRansformer (DETR) [364] significantly outperforms competitive baselines in both object detection and semantic segmentation. LeViT [365] outperforms existing benchmarks with respect to balancing the accuracy and training speed. Image GPT [149] is inspired by a sequence transformer in NLP, which can compete with several self-supervised benchmarks on ImageNet. On the basis of this research, DeepViT [366] explores a deeper architecture to improve performance consistently by making the transformer go deeper. Moreover, many researchers try to apply the transformer to more specific tasks. Pyramid Vision Transformer (PVT) [367] introduces the pyramid structure to overcome the difficulties of porting the transformer to various dense prediction tasks, and achieves the SOTA performance on major benchmark datasets. M3DeTR [368] is a novel research on multi-representation, multi-scale, and mutual-relation 3D object detection with transformers. Medical Transformer (MedT) [369] has focused on medical image segmentation and outperforms previous CNN-based and transformer-based architecture. In conclusion, the transformer has become a novel and popular research area in CV and its performance is proved by many existing works.

1.3 B.3 Traditional graph learning

GL aims to embed the graph as a low-dimensional representation while preserving the desired properties of the original graph data. Classical GL methods are usually implemented using statistical methods or artificially designed components.

Dimension reduction As a commonly used method in feature engineering, dimension reduction aims to reduce the dimension of high-dimensional attribute graph data into a lower-dimensional representation. In GL, it highlights the remaining information at the cost of losing part of the attributes. According to different dimensionality reduction strategies, such methods can be classified into two types. The first type is subspace learning under the linear assumption. Based on the assumption that the principal components [370] related to the larger variance represent important structural information, and those smaller variances represent noise, principal component analysis calculates a low-dimensional representation that maximizes the variance of the data. Linear Discriminant Analysis (LDA) [371] achieves dimension reduction by maximizing the ratio of inter-class scattering and intra-class scattering to obtain a linear projection matrix. Multi-Dimensional Scaling (MDS) [372] is a distance-maintaining manifold learning method. It produces a mapping in a lower dimension to preserve dissimilarities between nodes as much as possible. The second type is nonlinear dimension reduction, which aims to automatically learn nonlinear topology to achieve manifold learning. Isomap [373] first constructs a neighborhood graph on the manifold and calculates the shortest path between pairs of nodes, and then uses MDS to construct a low-dimensional embedding. Locally Linear Embedding (LLE) [374] first allocates neighbors for each node. Then, it calculates the weighted \(W_{i,j}\), the best linear reconstruction feature \(X_{i}\) from its neighbors. Finally, calculate the low-dimensional embedding for the optimal reconstruction of \(W_{i,j}\).

Matrix factorization Greatly influenced by the idea of dimension reduction, the models based on matrix factorization emerged in the early research of GL. Such models aim to reconstruct the adjacency matrix of the graph to achieve dimension reduction while maintaining structural information. Although these models have significant limitations, in fact, their ideas still inspire many current studies. Depending on how the matrix is constructed, such methods often append specific constraints. Graph Laplacian eigenmaps [375] minimizes a loss function to ensure that nodes close to each other on the manifold are mapped into the low-dimensional space and still maintain the local distances. Node proximity matrix factorization [376] minimizes the objective function \(|W-YY^{cT}|\) through matrix factorization to approximate the proximity of nodes in the low-dimensional space, where Y and \(Y^{c}\) are the embeddings for nodes and context nodes, and W is the default node proximity matrix. GraRep [377] aims to preserve the high-order proximity of graphs in the embedding space, thus it derives a k-th order transition matrix, \(A^{k}\), by multiplying the adjacency matrix to itself k times. The transition probability from node \(v_{i}\) to node \(v_{j}\) is the entry in the i-th row and j-th column of the k-th order transition matrix, i.e., \(p_{k}(v_{i}|v_{j})=A^{k}_{i,j}\). Then GraRep defines the loss function using the skip-gram model and negative sampling. To capture the high-order proximity between node pairs, HOPE [378] preserves asymmetric transitivity in approximating the high-order proximity. Specifically, the goal of HOPE is to minimize the objective function \(||S-WC^{T}||^{2}_{F}\), where the elements \(s_{i,j} \in S\) represent a certain edge feature (e.g., Katz index, the Rooted Page-Rank, the Common Neighbors, and the Adamic-Adar) between the corresponding node pairs \((v_{i}, v_{j})\), W is the node representation matrix, and C is the embedding of the node as the context. To reconstruct the matrix S more simply and elegantly, HOPE proposes to obtain W and C directly based on the low-rank singular value decomposition (SVD).

Graph kernel The kernel method is an important algorithm in pattern recognition and machine learning. Its basic idea is to give the graph embedding \(x \in X\) in the original low-dimensional space X, and maps the embeddings to a high-dimensional feature space H through a nonlinear function \(f^{ker}\). Then the nonlinear problem in X can be solved by constructing a linear algorithm in H. There are two main types of kernel methods on graph data. The first type uses the embedding method to convert the graph data into vectorial representation, and then directly implements the application based on the kernel function. However, due to the loss of mass graph structure information when transforming graphs into vectorial representation, such methods do not perform well in real scenarios. The second type of method introduces the graph kernel function to solve this problem. Based on retaining the advantages of the original kernel function, it directly represents the structural information of the graph data in the high-dimensional Hilbert space. The definition of the traditional method of graph kernel comes from R-convolution. According to the difference between the contrast substructure and the decomposition method of the graph structure, a large number of methods based on graph kernel have been proposed. For example, the work of [379, 380] proposed a random-walk kernel based on calculating the number of common synchronization between two graph structures, To reduce the computational complexity and optimize the random walk strategy, a graph kernel based on comparing the shortest path information between two graph structures is proposed. To capture more complex topological information, the Weisfeiler-Lehman subtree graph kernel is proposed, which is based on a one-dimensional Weisfeiler-Lehman isomorphism test algorithm to find isomorphic subtree structures in a bunch of graph structures [381].

C PFMs theory

Since pretraining has received great attention from the research community, the investigation in the theory-backed explanation is similarly eye-catching. During the unsupervised pretraining era before SSL, Erhan et al. [382, 383] shed some light on the theoretical explanation for the confirmation and clarity of learning difficulties. [382] researches the influence of pretraining with respect to architecture depth, model capacity, and the number of training samples, and demonstrates the robustness of pretraining from the perspective of both the optimization and the regularization. [383] further prove the regularizer role of the unsupervised pretraining in the downstream supervised tasks.

1.1 C.1 Different perspectives

Pretext tasks. [384] posits a mechanism based on approximate conditional independence (CI) to connect pretext and downstream task data distributions, which suggests that pretext tasks can self-supervisedly learn the representations from unlabelled data that reduce the sample complexity of downstream supervised tasks. The experiments both on CV and NLP tasks support this theory. Representation Learning via Invariant Causal Mechanisms (ReLIC) [181] also provides a theoretical understanding from the perspective that the explicit invariance constraints across augmentations can yield improved generalization guarantees.

Multi-view redundancy. From the perspective of a multi-view setting, [385] understands contrastive learning as exploiting multiple views of data for representation learning. This theory provides a theoretical analysis that the linear functions of these representations from pretraining are still competitive compared with the non-linear optimal predictor of the label. In other words, the linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the different views provide redundant information about the label.

1.2 C.2 Different categories

Contrastive learning. Although experimental results show us that previous designs such as contrastive loss or momentum updating can produce impressive performance in SSL. However, one of the most important questions that remain in SSL is why these methods can maintain representation consistency during the pretraining process. A naive view is the minimization between positive pairs can boost invariance learning, while the maximization between negative pairs contributes to avoiding representational collapse. [386] shows that contrastive learning can achieve competitive bound via intra-class concentration, thus leading to the reduction of sample complexity on downstream tasks from the benefit of transferred representations. This research also provides a framework that can be utilized both on the guarantees of the quality of learning representations during the pretraining phase and the future assumptions added to the framework that allow tighter guarantees.

Non-contrastive learning. While contrastive learning shows an effect by capturing the similarity and dissimilarity among the unlabelled examples, and further converging to an average local optimum which represents the general representations, recent non-contrastive SSL methods such as BYOL and SimSiam also shows the SOTA performance without the design of comparison between negative pairs. Based on the analysis of the eigenspaces, Tian et al. [182] study the behavior of non-contrastive SSL training and prove that the effects are from both the predictor and stop-gradient signal. Based on this theory, a novel and simple DirectPred method is proposed as a by-product of this theoretical exploration.

D Pretext task taxonomy on CV

Pretext tasks are always designed to use pseudo labels generated from the data itself to pretrain the proxy model. There are five categories of pretext tasks for self-supervised: (1) generation-based methods; (2) transformation-based methods; (3) context-based methods; (4) semantic-based methods; (5) view-based methods.

Generation-based methods This type of method is GAN-based in the deep learning era. For image generation, there are several applications including image colorization [138, 387], image super-resolution [388], image editing [389], context encoders [137], image-to-image translation [352], etc. On the other hand, video generation tasks contains future prediction [145], video action recoginition [241], video generation [390, 391], and video representaion [392].

Transformation-based methods Transformation is a typical technology that serves as a data augmentation method to enlarge the training dataset in traditional DL. However, if transformations of the same image are labeled as positive samples and others as negative samples, this pretext task can be used for self-supervised pretraining [166]. Popular transformation in self-supervised learning (SSL) contains color transformation (such as Jitter, Gaussian blur, and adjusting brightness) and geometric transformation (such as flipping, cropping, scaling, and rotation).

Context-based methods Basically, the design and construction of many artificial tasks, such as solving Jigsaw puzzles [140], comparing context similarity, and discriminating sequence order. Solving Jigsaw puzzles is defined as identifying the correct position of patches from an image. This task can help the model to learn an encoder for transfer learning [141, 393], and the feature representations are effective after the pretrained dataset is big enough. In addition, the design of video Jigsaw is also proposed for unsupervised learning [394]. Differently, context similarity tries to label the patches from the same images as positive samples and others as negative samples, then use a predefined similarity function to scale the distance between different pairs [49].

Semantic-based methods Semantic-based methods contain object detection, semantic segmentation, and depth prediction. These tasks also involve pretext tasks because their pixel-based labels can learn a more robust feature representation than simpler tasks. These pre-text tasks always establish on video dataset [395, 396].

View-based methods This type of method contains both single-modal data and multi-modal data. For the single-modal data, the original data is treated as the anchor and different viewpoints generate its positive pair samples. Sometimes the time slices in sequence-based data are treated as negative pairs because the scene is changed as time goes [397]. In addition, multi-modal data is usual in view-based methods, which are also called cross-modal-based methods here. Such as audio-video cooperative learning [398], RGB and optical flow cross-modal distance training [250].

E Evaluation metrics

Classification task The classification task, according to a labeled training document, determines the relationship between document features and document categories. The learned relationship model is then used to determine the category of new documents.

Accuracy and error rate The key metrics for a text classification model are Accuracy and Error Rate. The terms Accuracy and Error Rate are defined as follows:

where \(\textrm{TP}\) and \(\textrm{FP}\) denote true positive and false positive, \(\textrm{TN}\) and \(\textrm{FN}\) stand for true negative and false negative.

Precision, Recall and F1 Regardless of the standard type and error rate, there are very important metrics used for unbalanced testing sets. These metrics are similar to the concept of the class label in the testing samples. F1 is defined as the harmonic average of Precision and Recall. Thus, Accuracy, Recall, and F1 can be represented as:

When the accuracy, F1, and recall values hit 1, the desired results are obtained. On the other hand, when the values turn 0, we get the worst consequence. For the multi-class classification task, the precision and recall values of each class can be determined independently, and then the individual and overall performance can be analyzed.

Micro-F1 The Micro-F1 [399] is a metric that measures all labels’ overall accuracy and recall. We denote \(Micro\text {-}F1\) as:

where \(T P_{t}\) and \(F P_{t}\) mean true and false positive of the t th label on a text.

\(Macro\text {-}F1\) The \(Macro\text {-}F1\) calculates the average F1 of all labels by giving equal weight to them. \(Macro\text {-}F1\) is denoted as:

where \(T N_{t}\) and \(F N_{t}\) represent true and false negative of the t th label. \(\mathcal {S}\) stands for the label set of all samples.

Mean reciprocal rank (MRR) The MRR is commonly used to evaluate the performance of ranking algorithms on Question Answering (QA) and Information Retrieval (IR) tasks. MRR is represented as

where \({rank}_{i}\) is the ranking of the i th ground-truth answer. The number of predicted labels on each text is denoted by Q. Moreover, there are some metrics, such as EM, Hamming-loss [400], P@K and NDCG@K.

Generation task Generation task uses LMs to predict the next most likely word or sentence based on input data.

Bilingual evaluation understudy (BELU) BLEU compares the generated sentences to the reference sentence and makes predictions using automatic machine translation algorithms. The language creation problem is also supported by deep learning technologies such as speech recognition, image caption generation, and text summarization. They can’t discover anything better, but it has a few advantages: it’s simple to comprehend, correlates well with human judgment, and is language-independent. As a bilingual evaluation aid, BLEU is mainly used to evaluate the quality of machine translation [401]. BLEU compares the degree of overlap between the N-gram in the candidate text and the N-gram in the reference text. The higher overlap indicates better translation quality. The formula for the computation is:

where N represents N-gram, BP is penalty factor, \(P_{N}\) is multivariate precision, and \(W_{N}=1/N\) is the corresponding weight of multivariate precision. r represents the length of the shortest reference translation, and c represents the length of the candidate translation, then the specific calculation method of penalty factor BP is as follows:

where \(l_{t}\) is the number of words in machine translation and \(l_{a}\) is the number of words in reference answer. The penalty factor is mostly used to penalize large gaps between machine and reference translations.

ROUGE (recall-oriented understudy for gisting evaluation) ROUGE stands for N-gram co-occurrence statistics, which are used in automatic evaluation methods. It is expanded on the similarity of N-grams, which means that an N-gram is a subsequence of the main document text in terms of N words. There are four types of ROUGE, including ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S. The first two are commonly used, and the N in rouge-N refers to N-gram, which is calculated similarly to BLEU, except BLEU is based on accuracy, while ROUGE is based on recall. L in ROUGE-L refers to the Longest Common Subsequence, which is calculated as the Longest Common Subsequence between the candidate abstract and the reference abstract. Thus, the longer the length, the higher the score, based on the F value. The calculation formula of ROUGE-N and ROUGE-L is mainly introduced. The calculation formula of ROUGE-N is as follows:

where N stands for N-gram, \(Count(gram_{n})\) represents the frequency of occurrence of an N-gram, and \(Count_{match}(gram_{n})\) represents the frequency of co-occurrence of an N-gram. The calculation formula of ROUGE-L is as follows:

where X is the candidate abstract, Y represents the reference abstract, LCS(X, Y) table indicates the length of the Longest Common Subsequence (LCS) of the candidate abstract and references abstract, M stands for the length of reference abstract, and N denotes the length of the candidate abstract. The ROUGE method is characterized by N-gram co-occurrence statistics, based on recall rate (ROUGE-N) and F-value (ROUGE-L). They are often used in text summaries. It is worth noting that ROUGE is word-based correspondence rather than semantic-based correspondence, but this can be mitigated by increasing the number of reference summaries.

METEOR METEOR, also known as an explicitly sorted translation evaluation metric [402], is an improved version of the BLEU standard that aims to address some flaws in the BLEU standard. Using WordNet to calculate matching relationships between specific sequences, synonyms, roots, affixes, and definitions improves BLEU performance and makes it more relevant to manual discrimination. The calculation formula is as follows:

where \(Pen=\gamma (\frac{ch}{m})^{\theta }\) is a penalty factor, which punishes the word order in candidate translation that is different from that in reference translation. ch refers to the number of chunks, which are clustered units of matched units adjacent to each other in both the candidate translation and the candidate reference translation. \(\alpha , \beta , \theta\) is the adjustable parameter, m is the number of unary groups that can be matched in the candidate translation, c is the length of the candidate translation, \(h_{k}(c_{i})\) is the number of occurrences in candidate translations \(c_{i}\), and \(h_{k}(s_{ij})\) is the number of occurrences in reference translations \(s_{ij}\).

Perplexity Perplexity is also called the degree of confusion [403]. Its core idea is: first, according to the testing sentence, learn an LM P. Then, according to the LM P, the score of the optional sentence is calculated. Finally, the above scores are standardized according to sentence length. The calculation formula is as follows:

where W is the candidate translation, M is the length of the candidate translation, P is the LM obtained according to the reference translation, and \(P(w_{1}, w_{2}, \ldots , w_{M})\) is the score calculated by the LM for the candidate translation. The Perplexity assessment indicator is based on a LM. The lower the degree of confusion, the better the translation quality, which is often used in machine translation and LMs. Its disadvantages are as follows: the larger the data set is, the faster the degree of confusion decreases; the punctuation in the data will impact the PPL of the model; and the interference of common words.

F Datasets

1.1 F.1 Downstream tasks and datasets on NLP

There are many available data sets in the NLP domain, divided according to different tasks. It mainly comprises two categories: the task of classification of texts and the task of generating texts. The text classification tasks mainly include Sentiment Analysis (SA), News Classification (NC), Topic Labelling (TL), Natural Language Inference (NLI), Named Entity Recognition (NER), Question Answering (QA), Dialogue Act Classification (DAC), etc. The generation tasks mainly include text summaries and machine translation (Table 5).

Sentiment analysis (SA)

It consists of judging the emotional polarity and dividing it into several classes. Depending on the granularity of sentiments, the SA is divided into three categories: dichotomy (positive and negative), trichotomy (positive, negative, and neutral), and multiple categories. Here we introduce several datasets in detail.

Stanford sentiment treebank (SST) [474] The dataset is an extension of MR [475]. SST-1 is a version of SST. It is divided into five categories and the number of training texts and testing texts is 8544 and 2210, respectively. It also consists of 20 average tokens. The SST-2 [409] contains 9613 movie reviews including 6920 training texts, 872 development texts, and 1821 testing texts.

Semantic textual similarity benchmark (STS-B) [476] It is used in semantic textual similarity tasks organized in the SemEval context between 2012 and 2017 [477]. It consists of text from image titles, news titles and forums. On a scale of 1 to 5, STS-B displays the semantic similarity of two sentences. It includes 5749 training sets, 1379 development sets, and 1377 testing sets.

Multi-perspective question answering (MPQA) [478, 479] This is an opinion dataset which has two categories. It contains 10,606 sentences from various news sources that have been manually annotated for opinions and other private states. It is worth noting that there are 3311 positive articles and 7293 negative articles, having no labels for each article.

IMDB reviews [480] The dataset is the world’s most authoritative source for binary sentiment classification of film reviews. The number of content in each class is the same and it can be divided into training and testing sets whose number of comments is 25,000 on average.

News classification (NC)

As one of the most vital information sources, news content exerts a critical effect on people. The NC facilitates users to acquire essential knowledge in real time. Its applications mainly include news topic identification and recommendation of relevant news based on user interests. Here we introduce several datasets in detail.

20 Newsgroups (20NG) [481] 20NG is a text dataset derived from newsgroups. There are 20 classes with the same number of articles per class, including 18,846 articles in total. The average number of tokens is 221.

AG news [424, 482] This is an academic news search engine, which is divided into four categories. It contains news headlines and introductions. It includes 120,000 training texts and 7600 testing texts. The number of average tokens is 45/7.

R8 and R52 [483] They come from Reuters [484]. R8 contains 8 classes consisting of 66 average tokens and includes 2189 and 5485 testing and training courses. There are 52 classes in R52, which consists of 70 average tokens. It is divided into 6532 and 2568 training and testing texts.

Topic labeling (TL)

The task mainly obtains the meaning of the file by defining complex file themes. It is a critical component of topic analysis technology, which aims at simplifying topic analysis by assigning each article to one or more topics. Here, we introduce a few in detail.

DBpedia [485] It is a large-scale multilingual knowledge base generated by Wikipedia’s most commonly used information boxes. It releases DBpedia every month, adding or removing classes and attributes in each version. The most popular version of DBpedia has 14 categories, separated into 560,000 training data and 70,000 testing data. The number of average tokens is 55.

Ohsumed [486] This is a biomedical literature database. The number of texts is 7400. It has 23 cardiovascular disease categories and consists of 136 average tokens. All texts are medical abstracts that are categorized into one or more classes.

Yahoo answers (YahooA) [424] The dataset is a topic labeling task having 10 categories. The number of average tokens is 136. There are 140,000 training data and 5000 testing data. Each text in YahooA has question titles, question contexts, and best answers.

Natural language inference (NLI)

This task is used to forecast whether the meaning of a text can be inferred from another. Interpretation is a broad form of NLI. By comparing the semantic similarity of sentence pairings, it determines whether a sentence is the interpretation of another one. Here we introduce several primary datasets in detail.

The Stanford natural language inference (SNLI) [430] It is commonly used in NLI takes. It contains 570,152 human-annotated sentence pairs, which are annotated with three sorts of relationships: neutral, derived, and conflicting. Multi-genre Natural Language Inference (MNLI) [487] has 3 categories and consists of 430,000 sentence pairs annotated with textual information, which is usually used in textual inference tasks. Question Natural Language Inference (QNLI) [488], whose task with 2 classes is to determine whether a given text pair is a question-answer. Winograd Natural Language Inference (WNLI) [489] which consists of 2 categories is a data set that captures the standard reference information between two paragraphs.

Microsoft research paraphrase (MSRP) [435] The dataset contains sentence pairs for the text-similarity task, including 1725 training and 4076 testing sets. A binary label annotates each pair, discriminating whether they are paraphrases.

Sentences involving compositional knowledge (SICK) [434] It includes nearly 10,000 English sentence pairs, marked with similarity, and the scale range is 1–5. It has neutral, entailment, and contradictory three categories.

Named entity recognition (NER)

This is a fundamental task of NLP to identify people, places, organizations, and other entities in text. It is a crucial primary tool for many NLP tasks, including information extraction, question answering, semantic parsing, machine translation, etc.

CoNLL 2003 [303] It consists of newswire text from the Reuters RCV1 corpus. It contains four different entity types (Location, Organization, Person, and Miscellaneous) and includes 1393 English news articles, and 909 German news articles.

OntoNotes 5.0 [13] The dataset consists of 174,5K English, 900K Chinese, and 300K Arabic text data. It comes from telephone conversations, news agencies, radio news, radio conversations, and blogs. It has 18 entity classes containing 11 types, seven values, and 2,945,000 text data.

MSRA [440] This is a Chinese dataset that is obtained from the news domain. It has three types of entities and is used as a shared task on SIGNAN back in 2006.

Question answering (QA)

There are two types of QA systems: the extraction guidance system and the generation guidance system. The extractive QA can be regarded as a particular case of text classification. Here we detail several datasets.

Microsoft research paraphrase corpus (MRPC) [490] It contains 5800 sentence pairs extracted from Internet news, and the task type is similar to the QQP dataset. Sentence pairs are derived from comments on the same news item and determine whether the two sentences are semantically the same. The assessment criteria were classification accuracy and F1 score.

Stanford question answering dataset (SQuAD) [303] This is a large-scale machine-reading comprehension dataset that contains two tasks. SQuAD 1.1 [488] provides questions and corresponding answers, and the data set contains 100,000 samples in total, while SQuAD 2.0 [491] adds unanswered questions and expands the scale to 150,000.

RACE [492] The dataset has 5 categories, containing nearly 100,000 questions extracted from middle and high school English tests, with corresponding answers given by experts. The average length of RACE text is more significant than 300, which is longer than other reading comprehension data sets (such as SQuAD) sequences.

Dialog act classification (DAC)

The dialogue act is a specific verbal component, which marks the dialogue according to the meaning category of the dialogue. DAC categorizes tags according to the meaning of the dialogue to help understand the speaker’s intentions.

Dialog state tracking challenge 4 (DSTC 4) [451] It belongs to the dialog act classification task and mainly focuses on dialog state tracking on human-human dialogs. It is divided into 89 training classes and contains 24,000 training texts and 6000 test texts.

ICSI meeting recorder dialog act (MRDA) [452] It includes about 75 h of speech from 75 naturally occurring meetings among 53 speakers. The number of categories is 5, and it contains 51,000 training texts, 11,000 test texts, and 11,000 validation texts.

Switchboard dialog act (SwDA) [493] The dataset extends the dialogue behavior label with rounds/discourses. The label summarizes the sentence structure, and relevant and pragmatic information of the relevant turn. The SwDA is split into 43 training classes and includes 1,003,000 training texts, 19,000 test texts, and 112,000 validation texts.

Text summarization

Text summarization is a summary of a given single or multiple documents. It is kept as concise as possible while ensuring that it reflects the critical content of the original document. It can be divided into extractive summarization and generative summarization. Extractive summarization is generated by extracting and splicing the critical sentences in documents. Generative summarization is generated by a model, which summarizes documents according to the required content expressed in documents.

NYT [455] The dataset comes from the corpus annotated by the New York Time. The named entities are annotated using the Stanford NER tool in conjunction with the Freebase knowledge base. It contains 9076 articles, with the remaining 100,834 divided into a training set (96,834 examples) and a validation set (4000 samples).

CNN/daily mail [457] It is used for the passage-based question-answering task, and it is popular in assessing ATS systems. The dataset consists of CNN/Daily Mail news stories paired with multi-sentence human-generated summaries. There are 287,226 training instances, 13,368 validation instances, and 11,490 testing instances in total.

Gigaword [464] This is a data set of English news chapters consisting of nearly 950 pieces. Headlines – stories from multiple sources, including the New York Times – include some articles with a one-sentence, short news feed.

Machine translation (MT)

It refers to the task of translation from one language to another with its semantic equivalence by a computer. There are three categories, rule-based machine translation, statistics-based machine translation, and neural network-based machine translation.

WMT14 [465] It is a grouping of datasets used in the Ninth Workshop on Statistical Machine Translation shared tasks, including a news translation task, a quality estimation task, a metrics task, and a medical text translation task.

WMT16 [466] This dataset is a grouping of datasets used in the First Conference on Machine Translation shared tasks. It has ten shared tasks, including a news translation task, an IT domain translation task, a biomedical translation task, an automatic post-editing task, a metrics task, a quality estimation task, a tuning task, a pronoun translation task, a bilingual document alignment task, and a multimodal translation task.

WMT17 [465] The dataset includes three MT tasks (news, biomedical, and multimodal), an automatic post-editing task, a quality estimation task, a task dedicated to the training of neural MT systems, a task on bandit learning for MT, an automatic post-editing task, and a metrics task.

WMT18 [468] It mainly features six shared tasks: a news translation task, a biomedical translation task, an automatic post-editing task, a metrics task, a quality estimation task, and a multimodal translation task. Participants must evaluate their approaches to the machine translation topic using the standard data sets created for the shared tasks.

Dialogue

As an essential way of man–machine interaction, the dialogue system offers a wide range of applications. The existing dialogue systems can be grouped into task-oriented dialogue systems and non-task-oriented dialogue systems from application scenarios. Among them, the non-task type of conversation system can also be called a chatbot.

DSTC2 [471] This is a multi-round dialogue data set of restaurant reservation fields, including 1612 training data, 506 verification data, and 1117 test data. It allows the user’s goals to change compared to DSTC1. DSTC2 is also richer in terms of the conversation state representation, including the slot value pairs of the user’s targets and the ways to find them.

MWOZ [471] It contains 8420/1000/1000 conversations for training, validation, and test sets, respectively. It contains 30 pairs in seven domains being a multi-domain fully-labeled corpus. Every sample includes a goal, multiple user and agent utterances, and annotations regarding slot values.

Out-of-scope (OOS) [471] The dataset includes 15,100 training, 3100 validation, and 5500 test sets, respectively. It contains 151 intent classes, containing 150 in-scope and one out-of-scope intent. The out-of-scope intent indicates that a user utterance failed to classify to given predefined objectives.

1.2 Downstream tasks and datasets on CV

The datasets in CV mainly contain three types from the perspective of tasks: classification, detection, and segmentation. The popular datasets are concluded in Table 6, and some infrequently mentioned datasets in long tails are discussed in the text.

Classification

In this part, we first cover the popular large-scale datasets used frequently in both the pretext and downstream tasks. Then the domain datasets only used for the downstream tasks are unfolded.

MNIST [497] It’s a collection of handwritten digits that includes 60, 000 samples in training and 10, 000 in testing. The images are fixed-size with \(28\times 28\) pixels. The pixel values are from 0 to 255.0 in which pixel values smaller than 255.0 can be understood as background (white) and 255 means foreground (black). The labels are from 0 to 9 and only one of these digits exists in an image. Both traditional and deep learning methods are based on this most popular dataset despite advanced methods showing perfect results. Thus, Geoffrey Hinton has described it as “the drosophila of machine learning”.

Street view house numbers (SVHN) [498] In the domain of digit numbers, it collects real-world digit numbers from house numbers in Google Street View images. It includes 73, 257 digits for training, 26, 032 digits for testing, and 531, 131 additional. All of them are \(32\times 32\) color images with both class labels and character-level bounding boxes.

CIFAR [499] As more advanced methods show perfect results on the simple datasets, more sophisticated datasets such as CIFAR-10 and CIFAR-100 are conducted. These two datasets are closer to the real-world object. The CIFAR-10 contains 50,000 training images and 10,000 testing images, with 6000 images per class and \(32\times 32\) pixels in each RGB color image. The CIFAR-100 is similar to the CIFAR-10 but with more detailed label information. There are 100 classes containing 500 training images and 100 testing images in each class. In addition, these 100 “fine” classes are grouped equally into 20 “coarse” classes. Researchers can adapt it to suitable learning methods.

STL-10 [500] Inspired by the CIFAR-10 dataset, STL-10 is another \(96\times 96\) color image dataset containing similar 10 real-world classes. Each class has 500 training images and 800 testing images. The biggest difference is that STL-10 has 100, 000 unlabeled images for unsupervised learning. More construction information can be seen in [501].

Caltech-101 [502] It collects roughly \(300\times 200\) color images of objects belonging to 101 categories, with 40–800 images per category and 50 on average. The outlines of the objects in the pictures are annotated for the convenience of different learning methods.

ImageNet [312] This is one of the most popular and large-scale datasets on computer vision. It is built according to the hierarchical structure of WordNet [503]. The full ImageNet dataset contains 14, 197, 122 images and 21, 841 synsets indexed, attaching on average 1000 images to demonstrate each synset. The most frequently-used subset of ImageNet is the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset from 2010 to 2017, containing tasks of classification, localization, and detection. The number of samples in training and testing datasets and the labels of images are determined by the specific task, more details are seen in [504].

HMDB51 [505, 506] In addition to the popular MNIST, there still exist many domain datasets used for the downstream tasks in the classification problem. HMDB51 is an action video database for a total of 7000 clips in 51 action classes. It contains five types of facial actions and body movements.

UCF101 [507] It is another action video data set designed for more realistic action recognition. It is an extension of the UCF50 [508] data set containing only 50 action categories with 101 action categories, collected from YouTube. What makes it a famous recognition dataset is the workshop in ICCV13 with UCF101 as its main competition benchmark.

Food-101 [509] This is a real-world food dataset of 101 food categories, with 750 and 250 images per class in training and testing dataset respectively.

Birdsnap [510] It is a fine-grained visual categorization of birds on a broad scale, with bounding boxes and the locations/annotations of 17 parts in the object. It contains 49,829 images of the 500 most common species in North America, with each species containing 69–100 images and most species having 100. In addition, some images are also labeled as male or female, immature or adult, and breeding or non-breeding plumage.

SUN397 To target the scene categorization, the extensive Scene UNderstanding (SUN) database [511, 512] fills the gap of the existing dataset with the limited scope of categories. This database has 899 categories and 130,519 images, and only images with more than \(200\times 200\) pixels were kept. SUN397 is a more well-sampled subset that maintains 397 categories with at least 100 images per category, in which other categories containing relatively few unique photographs are discarded.

Places205 Places205 [513] dataset is another large scale scene dataset consists of 2,448,873 images from 205 scene categories.

Cars [514] The dataset in the domain of cars contains 16,185 color images of 196 classes (at the level of Make, Model, Year) of cars. For convenience, this dataset is split into training and testing sets in roughly equal quantities.

Aircraft [515] It is another fine-grained visual classification designed for aircraft (also known as FGVC-Aircraft). A popular form of this dataset is the fine-grained recognition challenge 2013 (FGComp2013) [516] ran in parallel with the ILSVRC2013. There exist four-level hierarchies: Model, Variant, Family, Manufacturer, from finer to coarser to organize this database. The more detailed information is shown in [517].

Pets [518] It represents The Oxford-IIIT Pet Dataset that collects 37 pet categories with roughly 200 images per category. All images have an associated ground truth annotation of breed for classification, head ROI for detection, and pixel-level trimap for segmentation.

Flowers [519] Similarly, Flowers is another domain dataset in flowers also collected by Oxford; it contains Oxford-17 Flowers of 17 categories and Oxford-102 Flowers of 102 categories.

Describable textures dataset (DTD) [520] This is an evolving collection of textural images in the wild, which consists of 5640 images of 47 categories, with 120 images per category.

iNaturalist2018 [521] It is a large-scale species classification competition conducted on the FGVC5 workshop at CVPR2018. This dataset contains over 8000 species categories, with more than 450, 000 images in the training and validation dataset collected from iNaturalist [522].

JFT-300 M [523] JFT-300 M is an internal Google dataset introduced by Sun et al [523] and well-known from ViT Model [40]. It is labeled by algorithms that utilize human-computer communications and target classification tasks. This dataset finally contains 300 M images with over 1000 M labels, thus leading to the multiple labels attached to this large-scale dataset.

Detection

The detection is a popular task in the CV, and almost all the research is conducted on COCO and PASCAL VOC datasets.

COCO [339] This is a large-scale data set for object detection, segmentation, and caption; it contains 330, 000 RGB images, with more than 200, 000 labeled. There are 1.5 million object instances of 80 object categories involved. Thus, it is one of the most popular benchmark data set in detection and segmentation in parallel with the following PASCAL VOC.

PASCAL VOC [524] From 2005 through 2012, the dataset has run challenges assessing performance on object class recognition and has provided standardized image datasets for object class recognition. The main datasets used in self-supervised learning are VOC07, VOC11, and VOC12. Main competitions in VOC07 [525] contain classification and detection tasks; both of them consist of 20 objects and contain at least one object in each image. Thus, it is common to use VOC07 to serve as the downstream task for the detection.

Segmentation

The segmentation is a semantics-based pixel-level classification. These datasets are difficult to obtain and annotate, thus they are always used as a downstream task.

VOC11 [526] & VOC12 [527] Both VOC11 and VOC12 contains classification, detection, and segmentation tasks in the main competition, thus leading to the common use of downstream task for the segmentation.

ADE20K [528, 529] It collects 27, 574 images from both the SUN and Places205 databases, in which 25, 574 for training and 2000 for testing. All 707, 868 objects from 3688 categories existing in images are annotated. Especially, this dataset contains 193, 238 annotated object parts and parts of parts, and additional attributes, annotation time, depth ordering for the benefit of the research community.

NYU-Depth V2 [530] This is a dataset consisting of images and video sequences from 464 indoor scenes that are recorded by both the RGB and Depth cameras from 3 cities. It contains 1449 images with the ground truth of depth, and the original RGB values are also provided. In addition, there are 407, 024 new unlabeled frames and additional class labels for the objects in images.

Cityscapes [531, 532] It is a dataset of urban street scenes from 50 cities with the ground truth of semantic segmentation. The main instances are vehicles, people, and construction. The high-quality dense pixel annotations contain a volume of 5000 images. In addition to the fine annotations, coarser polygonal annotations are provided for a set of 20, 000 images. Moreover, the videos consist of not consistent images with high-quality annotations, and these annotated images with consistently changing views are provided for researchers.

LVIS [533] It is a dataset for large vocabulary instance segmentation. It features that (1) a category or word in one image is related to the only segmentation object; (2) more than 1200 categories are extracted from roughly 160,000 images; (3) long tails phenomenon exist in these categories; and (4) more than 2,000,000 high-quality instance segmentation masks.

Densely annotated video segmentation (DAVIS) [534] It is a video dataset designed for the in-depth analysis of the SOTA in video object segmentation, in which DAVIS 2017 [535] contains both semi-supervised (human-guided at the testing time) and unsupervised (human non-guided at test time) video sequences with multiple annotated instances.

Others

There are many datasets designed for special visual tasks such as inpainting. In addition, this part covers the data collection in the wild.

Paris StreetView [536] The dataset is designed for image inpainting task, which contains 14, 900 training images and 100 testing images collected from street views of Paris. This dataset is collected from Google Street View and mainly focuses on the buildings in the city.

Moving-MNIST [537] Based on MNIST, it is a video dataset designed for evaluating sequence prediction or reconstruction, which contains 10,000 sequences. Each video is long of 20 frames and consisted of two digits (possibly overlapped) moving inside a \(64\times 64\) patch. The first benchmark is reported on [538] by the method of LSTMs.

Yahoo Flickr creative commons 100 million (YFCC100M) [539, 540] The dataset is the largest public multimedia collection that is allowed to search by users for their own targets; this dataset can browse both images and videos. It is free and for researchers to explore and investigate subsets of the YFCC100M in real time. Subsets of the complete dataset can be retrieved by any keyword search and reviewed directly. In addition, the text information attached to any image or video is abundant, such as containing location information and user tags. Briefly, it is more a multimedia library than a domain dataset.

Data in the wild More generalized dataset concept in the self-supervised learning era is composed of multimedia websites, APP, or search engines such as Instagram, Flickr, Google Images, etc. I think pictures in the wild will play a major role in the future study of CV because of the quantity of data, the computation source, and the learning power of PFM.

1.3 F.3 Downstream tasks and datasets on graph

The purpose of the pretraining graph model is to improve the performance of downstream tasks. According to the different analysis objects of the downstream tasks, they can be divided into nodes, edges, and graphs. Meanwhile, the PFMs of GL have been widely used in a mass of fields. In this section, we combine the downstream tasks to conduct statistics on the pretraining datasets and the downstream task datasets.

Node-level tasks

Nodes are the most basic element of the graph, so lots of downstream tasks mainly focus on the analysis of nodes.

Node classification Node ClassiFication (NCF) is one of the most prevalent graph-based tasks, which has important analytical value in most of the different types of graph data. Different from the pseudo-labels assigned to nodes in the graph in self-supervised methods, the labels in NCF often come from external information such as manual annotation. Based on Definition 7 and 8, NCF can be divided into two types: transductive and inductive according to the visibility during training, verification, and testing. In addition, the result of NCF can be single-label or multi-label according to the mutual exclusion of labels. The statistical results of common NFC datasets are shown in Table 7.

Node clustering The goal of Node ClusterIng (NCI) is to divide a graph into different classes or clusters according to a certain standard so that the correlation of nodes in the same cluster is as large as possible, and the irrelevance of nodes that are not in the same cluster is also minimized. Although in the above-mentioned pretraining tasks, NCI is used as a pretext task has appeared, NCI can still test pretraining graph models based on other pretext tasks.

Top-K search The goal of task Top-K Search (TKS) is to search the K nodes with the highest predefined associations for a given node in the graph. Usually, TKS is used for search tasks such as recommendation and alignment. The detailed statistical results of the datasets are shown in Table 7.

Link-level tasks

The edge is also an important part of the graph structure, which associates independent nodes and is the key to distinguishing graph data from non-relational data. Especially in some specific fields (e.g., molecules, proteins), edges contain real information, so there are various tasks related to edges.

Link classification Similar to the NCF, the Link Classification (LC) also assigns one or more labels to a given edge. In fact, in LC, the nodes at both ends of the edge are still taken into consideration.

Link prediction Link Prediction (LP) is a common graph task (e.g., knowledge graph). The goal of LP is to predict edges that are removed or may exist in the graph. Similar to NCI, LP is also one of the pretext tasks in self-supervised learning, and its statistic results as shown in Table 8.

Top-K recommendation Top-K Recommendation (TKR) is exactly the same as the definition of TKS, the difference lies in the sorting goal.

Graph-level tasks

The graph-level task generally focuses on the distribution of nodes, edges, and attributes in a given graph, in order to infer the possible properties of the entire graph.

Graph classification Graph Classification (GC) is commonly used in social, molecular, and protein graph data, which aims to predict the property of the given community, chemical compound, and protein. The statistic results as shown in Table 9.

Data source

The PFMs of GL have been widely used in a mass of fields. We will descript the details of the pretraining datasets and the downstream task datasets.

Citation and co-author network A citation is a basic local representation, whose structure reflects the citation relationships of papers in a research direction or field. Specifically, a citation network is a kind of relational data composed of research papers as nodes and citation relations as edges. Among them, the citation network used in the GL model usually comes from local samples of common citation databases, e.g., Cora, Citeseer, and PubMed, and serves as downstream tasks. Similarly, the co-author network is a dataset of scientific collaboration that corresponds to a researcher’s ego network, in which the researcher and their collaborators are nodes and an edge indicates collaboration between two researchers. According to different requirements of downstream tasks, such co-author networks can be used for various tasks, e.g., node classification and graph classification.

Molecular and protein network A molecular network usually refers to a compound composed of atoms and atomic bonds, and predicting the properties of the compound is usually regarded as a graph classification task. For example, MUTAG is a collection of nitroaromatic compounds whose goal is to predict their mutagenicity to Salmonella typhimurium. PTC uses a graph to show the structure of multiple compounds and aims to predict the carcinogenicity of different compounds in rats. The protein network is a collection of proteins classified as either enzymes or non-enzymes. The amino acids are represented by nodes, and two nodes are connected by an edge if they are less than 6 Angstroms apart.

Social and movie network The social network is the social-relational data in the real network environment, which usually represents the relationship between users or posts. For instance, Reddit is a graph dataset comprised of Reddit posts made in September 2014. BlogCatalog is a graph dataset that represents a network of social relationships between bloggers who are listed on the BlogCatalog website. The movie network is usually composed of actors and their co-occurrence participation in the movie. For example, IMDB-B is a movie collaboration data set that contains a large number of self-networks of actors who play movie roles in IMDB. Nodes in each graph represent actors/actresses, and if they appear in the same film, an edge connects them. These graphs are based on action and romance genres. The difference between IMDB-M and IMDB-B is that a node in the graph represents one or more actors.

Others Some of the rarer graph data are used to test the universality of the PFM, such as word networks (Wikipedia), book networks (Book-crossing), and airline networks (US-Airport). In addition, there are also some special graph structures adapted to specific models, such as spatiotemporal graphs (METR-LA).

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions