Content-based and Knowledge-enriched Representations for Classification Across Modalities: A Survey

In this section, we examine LLTM: representations that rely on matching templates on the input data [Bengio 2009], producing low-level responses. Given a representation template, we can discriminate between “local” (the template is applied on an input subsection) and global (the template spans the entire input instance) applications of the extraction process [Zhang et al. 2007; Mikolajczyk et al. 2005]. Local templates may regard the input as key-value pairs: keys locate distinct attributes (e.g., an individual word/ngram in text, a region/point of interest in images and audio) while values correspond to a magnitude of match or weight of the template in that location. Terms may be easily delineable in the source data (e.g., individual words in text, detected keypoints in images, specific temporal slice/peak in audio) or lack high-level semantics and an intuitive explanation—the latter can affect the interpretability of the representation and, down the line, of the entire classification pipeline [Došilović et al. 2018; Danilevsky et al. 2020]. Global features usually provide coarse information on a narrow view of the input—e.g., color distribution information in images, sentiment/grammaticality scores in text, SNR values for audio, and so on. [Oliva and Torralba 2006].

A popular representation is a Vector Space model (VSM) [Salton et al. 1975], which projects data into a vector \(v \in \mathbb {R}^d\) that can be manipulated with distance measures and Linear algebra constructs [Basseville 1989; Cha 2007] to process/compare instances. The Bag of Words/Features (BoW/BoF) [Salton and Buckley 1988; Sebastiani 2002] is a popular VSM that produces count-based weights for points/regions of interest in the input. BoF is a popular baseline for text, where semantically salient terms are easily identifiable and delineated by syntax and grammar. Common weighting schemes include boolean, term, and document frequency (BF, TF, DF), denoting presence, instance, and collection-level counts of single or n-tuples of terms (n-grams) in the text. The work in Badawi and Altınçay [2014] utilizes BoW features, along with an investigation on term weighting and selection in the binary classification of articles and biomedical documents. “Termset features” are introduced—i.e., tuples of document terms (e.g., words) where the feature activates if either one or both terms are detected. In Zhang and Wu [2015], the authors reduce sparsity by building an extension library, using bigram conditional probabilities in the text sequence and word-to-category similarity, based on word counts. Given a text, additional features are inserted with respect to their similarity score to the original feature set and introduced threshold heuristics. The extended feature set is then used to build the BoF representation.

The Term frequency-inverse document frequency (TFIDF) weighting scheme [Salton and Buckley 1988] normalizes term counts by DF weight, reducing the importance of tokens that occur too often in the document collection and behave like stopwords. In Trstenjak et al. [2014], the authors build a term-document frequency matrix to map articles to word TFIDF vectors, applying log-scaling and renormalization to improve robustness to varying document lengths. The authors in Sowmya et al. [2016] apply TFIDF to Wikipedia articles; document weights are pooled to category-level counts in order to build “centroid” vectors for each class, subsequently normalized by intra-class DF scores. In Thirumoorthy and Muneeswaran [2021], the authors propose feature selection schemes with respect of term and document frequency scores within and across classes. They use an evolutionary method, rich/poor population-based methods [Moosavi and Bardsiri 2019], and a classification-based fitness objective to optimize the final feature subset.

Contrary to text, BoF application in the visual domain is not straightforward: images lack clear semantic boundaries (“visual terms” are hard to delineate) and pixel-level approaches are intractable for most real-world tasks. Thus, visual LLTMs employ methods that apply (a) detection, i.e. locate regions of interest in the image [Tuytelaars and Mikolajczyk 2008], and (b) description, which applies low-level templates for building a representation for each such detected region.

A popular method is Scale Invariant Feature Transform (SIFT) [Lowe 2004; Younes et al. 2012]. It describes keypoints by normalized histograms of gradient orientations in the pixel intensities of a co-centric patch, that are largely invariant to shifts in illumination, viewpoint, rotation, and scale. SIFT is adopted for a variety of recognition tasks, e.g., detection/description in [Amato et al. 2015] for a local feature-based landmark classification along with other descriptors [Bay et al. 2008], while a similar approach extracts SIFT from a dense regular grid with patch overlaps [Bian et al. 2017].

Other methods extract fine-grained information suitable for texture, such as Gabor features [Mehrotra et al. 1992; Manjunath and Ma 1996], that involve the application of Gabor filterbanks. Gabor filters are applied on separate color channels in Risojević et al. [2011], using mean and stdev values over multiple scales and orientations, along with spatial envelope GIST features [Oliva and Torralba 2001]. Another approach is Local Binary Patterns (LBP) [Ojala et al. 2002], that extracts fine-grained, rotation-invariant binary-value histograms, via simple pixel-level comparisons between a center and its radial neighbors. LBP has been evaluated with different tasks, global/local contexts, resolutions, and focal configurations [Prasad and Mary 2019; Bian et al. 2017]. The authors in Ningtyas et al. [2022] utilize LBP along with Gray Level Co-occurence Matrix features (GLCM) to capture texture information for leaf classification.

ORB [Rublee et al. 2011] improves upon BRIEF [Calonder et al. 2010] features, adding “steering” mechanisms for rotation invariance and noise resistance, and is used in tasks like monument classification [Amato et al. 2015]. Further methods include Histograms of gradients (HoG) [Dalal and Triggs 2005], which computes distributions of pixel intensity gradient orientations, BRISK [Leutenegger et al. 2011], producing pairwise intensity comparison as binary features and KAZE [Alcantarilla et al. 2012], which applies nonlinear diffusion filtering for smoothing and multiscale operation; these are used for feature extractor/detection in various tasks [Prasad and Mary 2019; Amato et al. 2015].

In audio data, similar semantic ambiguity exists, as in the visual domain; However, given its one-dimensional and temporal structure, LLTM methods often apply simple features within time-segmented frames or in a global context.

A popular method is to capture statistical signal properties (e.g., mean, variance, extrema) and utilize the responses as feature vectors; for example, in Maršík et al. [2014] the authors consider RMS amplitudes as audio volume estimates, along with a self-similarity computation that counts the similar segments in a music piece. Further, in Zahid et al. [2015], signal sign change (zero crossing rate, ZCR) averages, short-time signal energy and periodicity analysis features are concatenated across audio window frames and used toward capturing repeated acoustic patterns.

Another approach switches to the frequency domain to mine low-level features from audio spectra. This is often achieved via Fast Fourier Transform (FFT) [Bracewell and Bracewell 1986]; FFT maps time-domain signals to frequency spectrograms via Fourier Analysis on small overlapping time windows. This is used in works like [Laurier et al. 2009], where the authors extract spectral statistic estimates like kurtosis, skewness, flatness, and flux for music emotion classification. Mel-frequency cepstral coefficients (MFCC) [Slaney 1998] involve multiple transformation, scaling, and normalization steps for short-term power spectrum description of an audio signal. It is used in studies like [Maršík et al. 2014], using mean and covariance statistics and [Zahid et al. 2015], along with spectral flux scores. Furthermore, Gammatone Cepstral Coefficients (GTCC) is a biologically-inspired modification to MFCC based on Gammatone filter functions with equivalent rectangular bandwidth bands; it is proposed and utilized in Valero and Alias [2012].

Some features target musicality/rythm-based information by using frequency binning and temporal progression statistics, or model psycho-acoustic phenomena by considering the operation of the human hearing system. For instance, timbral, perceptual and tonal information via dissonance and loudness measures are extracted in [Laurier et al. 2009], along with “danceability”, beats per minute (BPM), ZCR, and chord change features. The authors in Maršík et al. [2014] use musical LLTM like BPM, probabilistic estimates of chord root transitions, and musical keys. Furthermore, the work in Meister et al. [2022] uses an engineered feature bank composed of temporal, spectral, cepstral and tonal responses, which are ranked and evaluated for COVID patient classification.

In summary, this section showcased approaches that utilize LLTM features for classification; we now move on to methods that transform, manipulate and aggregate this information towards improving performance and tractability.

In this section, we explore approaches that rely on aggregating, combining and/or transforming lower-level representations to arrive at higher-level features, with respect to the abstractness and richness of the information encapsulated. Aggregation methods produce mid-level representations from low-level inputs, using engineered and/or learned functions rather than directly exploiting token-based statistics. In the scope of our analysis, these techniques correspond to the first attempts of improving the semantic content and richness of low-level representations by applying aggregation as a post-processing component. Contrary to low-level features, AGB methods usually build distributed representations [Hinton et al. 1984; Rumelhart 1986]: i.e., resulting semantics are spread or “distributed” over multiple dimensions in the embedding space [Bengio 2009], arriving at compact, robust features but sacrificing explainability. Dimensionality reduction is often facilitated by these approaches, mitigating the problem of the curse of dimensionality [Bellman 2013] while simultaneously maintaining or improving the expressive power of the output feature set.

Aggregation methods have presented different avenues for fusing LLTM features. Modality-specific delineation of semantically meaningful terms play an important role to the usefulness of different approaches.

For text data, local LLTM methods deal with distinct words, characters, and ngrams; as a result, the generated vocabulary can quickly scale to very large sizes, reducing system performance and accuracy [Bengio et al. 2005]. Thus, aggregation methods for text will often aim at shifting the representation from the vocabulary space to one more dense and compact, and simultaneously preserves the majority of the information content of its original counterpart.

A popular class of AGB methods is matrix decomposition techniques [Golub 1969], where a \(M \times d\) matrix of M, d-dimensional features are converted to a \(M \times k\) matrix, \(k \le d\). For instance, a popular method is Latent Semantic Analysis (LSA) [Deerwester et al. 1990], which produces document mappings to a set of k latent concepts. This is achieved by truncating the Singular Value Decomposition factors [Trefethen and Bau III 1997] of a term-document matrix, keeping the vectors with the k largest singular values. LSA is used in Zhang et al. [2011] to post-process TFIDF features and a multi-word term vector approach [Justeson and Katz 1995] for news articles categorization with linear SVMs. Additionally, Principal Component Analysis (PCA) [Jolliffe 2011] applies a change of base to the principal components of the original collection, that correspond to the covariance matrix of the original instances. Keeping a subset of k instances retains an optimal tradeoff with respect to variance retention and dimensionality reduction. PCA has been used in early works, for example in Li and Jain [1998], where term count features are post-processed with PCA and hierarchical clustering for news categorization with multiple classifiers. Additionally, in Zareapoor and Seeja [2015] both PCA and LSA are investigated for compression, reduction of complexity, and processing time of BoW features. Further, Latent Dirichlet Allocation (LDA) [Blei et al. 2003] generates topic models over Dirichlet priors, assuming distributions from instances to topics and from a topic to words. In Ye et al. [2017], the authors use TFIDF to model outlier words that are overlooked by LDA and include them for sentiment classification.

Other works adopt evolutionary methods to modify the representation; in Škrlj et al. [2021], weights for given diverse sets of input space features undergo mutation and crossover, while fitness is estimated by aggregate classification performance with multiple learners trained with SGD. Further, n-gram graph aggregation methods in Giannakopoulos et al. [2012] map documents to graphs, subsequently merged into class-level constructs. These are then compared with document-level graphs for categorization of the latter using different graph similarity measures.

In images, LLTM features generally produce non-scalar information, yielding large volumes of highly local responses. Many approaches use vector quantization [Gersho and Gray 1992] to fuse descriptor vectors and reduce workload to subsequent classification components, redundancy, and noise. Clustering methods are popular for this task; for example, Kmeans [Jain and Dubes 1988] merges populations of N local features to arrive at a pre-determined size of \(k \ll d\) visual clusters. These serve as an artificial vocabulary for a visual BoW analog, with distance-based assignments of local features to “visual words”. The bag vector is then used as a global image feature. KMeans vocabularies are used, e.g., in Zhu et al. [2019], over different combinations of SURF and BRISK for detection and description.

Modifications to the visual BoW include Spatial Pyramid Matching (SPM) [Lazebnik et al. 2006], which partitions and separately groups keypoints to preset image subdivisions. Further, locality-constrained linear coding (LLC) [Wang et al. 2010] enforces locality constraints on KMeans via regularization and modification the membership procedure, making instances being supported by multiple codebook bases for reduced reconstruction error and sparsity. LLC is evaluated by the work in Zou et al. [2016], aggregating dense SIFT descriptors in combination with global LBP features. Transformation-oriented methods like PCA are applied in the visual domain in Bodine and Hochbaum [2022], where a modification of Decision Trees [Lewis and Ringuette 1994] is proposed, which uses a one-dimensional, single-feature Maximum Cut split criterion, in conjunction with two localized PCA-based methods for transforming and mapping decision features. The vectors of locally-aggregated descriptors (VLAD) approach [Delhumeau et al. 2013] replace binary cluster assignments with concatenations of accumulated subtraction residuals. Another modification is Clustering with Fixed Centers (CFC) [Srivastava et al. 2019], over LBP features and SURF keypoints. Descriptors are grouped into bags assuming a fixed cluster center with respect to response maxima per LBP histogram bin, with combined category-level bags being used as the global feature. Further, fisher vector encoding [Perronnin and Dance 2007] uses visual vocabularies built with Gaussian Mixture Models (GMM) [Reynolds 2009], using the log-likelihood GMM gradients under the data are as encoded codebook features. A generalization to Riemannian spaces is proposed in Ilea et al. [2016], generated by Riemmanian Gaussian Mixture (RGM) models. The method uses region covariance descriptors (RCovD) as input features, built by covariance information from on sliding image patches.

In the audio domain, AGB approaches may exploit sequential signal interdependencies to pool together feature responses spatially/spectrally close to each other, along aforementioned methods of segmentation into “acoustic words”.

Aggregation via generative/probabilistic models has been used for classification, like GMMs in Lee and Ellis [2010]. There, MFCC inputs are combined with single and multiple-gaussian GMMs—pLSA is then applied on the built acoustic vocabulary. In the study of Kim et al. [2012] “Acoustic words” are formed from MFCC features via the LBG-VQ quantizer [Gersho and Gray 1992]. LDA is subsequently employed to generate audio-related topics over the audio vocabulary, with resulting weights serving as latent vectors for classification.

Like other modalities, matrix decomposition has also been applied for audio. In Baniya et al. [2014], the authors use PCA to reduce a feature set of spectral, dynamic, harmonic, and rhythm characteristics, along with higher order moments. Sparse coding (SC) [Lee et al. 2007] introduces sparsity bias to the encoding objective, in the form of a weighted L1 regularization term on the encoding vectors. A shift-invariant modification [Olshausen and Field 1996] proposed in Grosse et al. [2007] reconstructs the input signal to basis functions in all possible shifts to build temporally-invariant encodings. The authors apply computational efficiency improvements and compare the encodings with MFCC and raw spectrogram features. The approach in Zeghidour et al. [2021] applies a pipeline of filtering, pooling and compression/normalization composed of learnable steps, including Normalized Gabor convolutions, Gaussian low-pass filters and per-Channel Energy normalization [Wang et al. 2017], fed to MLPs for different categorization scenarios.

Having examined aggregation engineering of low-level features, the next section concludes the examination of knowledge-agnostic approaches by considering automatic DRL techniques.

Approaches covered so far used preconfigured templates locally interpolated on training data points, as well as manipulations of their responses in fixed, preconfigured steps. In this section, we focus on “deep” feature extractors, that aim at automatically learn multiple useful feature hierarchies from training data [Bengio 2009].

Typical representatives of this paradigm are graph-based computational models, such as the biologically inspired artificial neural networks (NNs) [Hubel and Wiesel 1962]. While deep generally refers to models that learn at semantically abstract, meaningful features, in the context of NNs, it can also refer to the size of the graph that implements the computation. Deep NNs are hierarchical models composed by multiple steps of nonlinear, learnable feature transformations. Contrary to previous paradigms, the line between features and classifiers is blurred: feature transformation, representation learning, and classification are simultaneously optimized in a holistic manner, learning the conversion of input data to prediction scores in an automatic, end-to-end fashion. This enables the design of efficient representation learners with fewer hard-coded, performance-critical parameters. As a result, deep approaches reflect further research efforts for improving representation semantics, diverging from rigorous feature and/or aggregation engineering and relying on a data-driven, automatic discovery of expressive features from scratch instead.

DRL methods generally exploit unsupervised pretraining [Bengio et al. 2007], fine-tuning, and transfer learning [Zhuang et al. 2020], with fitted model weights used as initializations for supervised training/fine-tuning for classification, or used directly as standalone features [Bengio et al. 2013; Pittaras et al. 2017].

Neural language models (NNLMs) [Arisoy et al. 2012] are popular approaches in text, applying distributional learning over big data to learning language structure and feature hierarchies via statistical learning of token distributions. NNLM-based pretraining and transfer learning DRL are an effective approach for deep feature extraction, as comparisons between NNLM-based methods, LLTM, and AGB techniques have indicated [Baroni et al. 2014],

Word embeddings are a popular example—for instance, Word2Vec [Mikolov et al. 2013] is pretrained via predicting center words given a context (CBOW), or the context, given the center word (Skipgram). Multiple studies exploit the effectiveness of transfer learning with Word2Vec features. In Yang et al. [2018], Word2Vec is pretrained over social media data with concatenated word vectors feeding a Convolutional Neural Network (CNN) with pooling components. Skipgram embeddings initialize the system in Liu et al. [2015], where LDA is used to build word and topic embeddings, exploring transfer learning variants to embed topics, word-topic pairs, or to concatenate topic and word vectors.

Some approaches utilize count-based inputs/seeds for faster operation and improved compatibility. In Chen et al. [2020], input TFIDF features are transformed into 2-dimensional matrices, fed to a CNN architecture equipped with pooling and fully-connected components. Other methods use transformers [Vaswani et al. 2017], relying on attention mechanisms [Bahdanau et al. 2015] to facilitate sequence modeling and learning, instead of recurrence or convolution. These methods have been shown to produce powerful, transferable representations, explored in studies like [Sun et al. 2019a], where the authors propose ways of fine-tuning BERT [Devlin et al. 2019] for efficient classification. Enhancements include additional pre-training, single/multi-task fine-tuning, and varying learning rates per transformer layer. In Pan et al. [2022], the authors introduce data augmentation by generating adversarial examples with FGSM [Goodfellow et al. 2015] by perturbing the word embedding matrix of a transformer, using the result for contrastive learning of noise invariant representations and improved performance on multiple NLU tasks including sentiment analysis.

In images, the large semantic gap between pixel-level and conceptual information has encouraged the design of DRL approaches. CNN architectures have been a very popular in this domain, with early successful approaches contributing to the rise in popularity of deep learning [Krizhevsky et al. 2012]. CNNs typically repeat layers of convolution, pooling and normalization, followed by fully-connected (dense) layers with dropout [Srivastava et al. 2014] and softmax normalization, and have been popular choices for large-scale image classification tasks [Russakovsky et al. 2015].

Different CNN topologies have been proposed to allow efficient training of larger models, e.g., residual connections, which preserve signals via additive identity operations and inception modules, which propose blocks of dimensionality reduction, pooling filtering and concatenation to improve scalability [Szegedy et al. 2015; He et al. 2016]. Additional approaches include the “densely connected” architecture, where layer outputs are linked to the inputs of all subsequent layers and vice versa, and residual connections replaced with feature concatenation. This topology is evaluated using the DenseNet model [Huang et al. 2017] on a wide range of image classification tasks. The model requires fewer parameters than ResNets, achieved by utilizing compression, bottlenecking and growth control techniques in the architecture. Further, in Touvron et al. [2022] image patches are processed via linear sublayers operating across patches and channels, equipped with residual connections and forming image-level predictions with average pooling.

After the success of transformers in NLP tasks, they have been utilized in the visual domain with similar success. In Dosovitskiy et al. [2021] a “Vision Transformer” (ViT) is used directly on raw patch sequences as well as CNN embeddings, using a classification head for Imagenet data categorization. The work in Liu et al. [2021] hierarchically merges image patches (like approaches in Lazebnik et al. [2006]) into larger visual tokens, computing self-attention to shifting windows rather than the entire token set, which achieves linear complexity and comparable performance. A similar approach is pursued in Chen et al. [2021], where image patches at different scales are encoded separately, and subsequently fused with cross-attention strategies of different granularity between encoded tokens of each scale.

Regarding deep approaches for audio, an established technique has been to first convert audio content into images and then apply a visual DRL pipeline. This conversion usually involves spectrograms, i.e., 2D time and frequency representations, produced by applying short-time Fourier transform [Sejdić et al. 2009] on segmented audio clips. Applying CNN-based architectures has been a broadly popular approach, in a manner similar to vision DRL models. For example, in Hershey et al. [2017], the authors perform a large-scale evaluation of popular CNN architectures designed for the visual domain [Krizhevsky et al. 2012; Simonyan and Zisserman 2015; Szegedy et al. 2015; He et al. 2016] on audio spectrograms, introducing variation to the amount of data and labelset size for each experiment. All visual models considerably outperform a dense NN baseline applied on raw spectrogram features on audio classification. in Nanni et al. [2021], ensembles of popular CNN models with different data augmentations (signal, spectral, visual, and so on.) are applied on spectrograms and evaluated at multiple benchmarks. Furthermore, the authors of Wang and Oord [2021] adopt a contrastive learning approach, using waveform and spectrogram-level representations in a siamese configuration. They employ CNNs and Residual architectures on MFCC/spectrogram and raw audio inputs for representation learning, applying fixed features on sound event classification tasks.

Approaches like deconvolution [Zeiler and Fergus 2014] reverses CNN computation to project filterbank activations back to the pixel space. This is exploited in Choi et al. [2016] towards reproducing audio corresponding to weights of trained CNN networks, aiming at improving explainability of learned features. They utilize an architecture of convolution, pooling, and fully-connected layers on audio spectrograms for music genre classification. Transformer models are also utilized, e.g., in Gong et al. [2021], where a transformer encoder pretrained on Imagenet is used on input audio on spectrograms, classifying output [CLS] embeddings to auditory classes and outperforming convolutional architectures.

A straight-forward avenue towards knowledge-rich approaches for classification is injecting knowledge alongside the content-based feature set C extracted from the input data. This involves modifying the collection of features fed to the learning model with high-level semantic, statistical, and conceptual information S. Such information is obtained by mining structured knowledge resources to extract knowledge relevant to the input instance, resulting in an enriched feature set \(E=f(C, S)\). The modification function f performs the fusion, e.g., by applying operations such as rule-based selection, concatenation, replacement, and so on.

A major component in IEM is selecting the resource that will provide the knowledge-based features, as these will merged to content-based features and be directly used by the downstream classifier.

Regarding text, resources such as Wordnet have been widely used for directly extracting conceptual/sense-level information from lexicalizations as well as exploiting hierarchical structures in its graph. Such schemes yield common features for different texts with similar semantics, improving generalization potential for the downstream learning machine. For example, the authors in Elberrichi et al. [2008] combine BoW with Wordnet sense statistics weighted via TFIDF. They explore different ways for mapping lexicalizations to senses (e.g., including expanding matches to hypernyms), concatenating the lexical and semantic channels and applying \(\chi ^2\)-based feature selection. A similar approach in Nezreg et al. [2014] improves upon Wordnet concept lookups by utilizing multi-word querying and POS information from Treetagger [Schmid 1994]. Further, conceptual mappings are expanded by considering hierarchical relations (e.g., hypernymy, antonymy) and different concept weighting schemes are investigated for the final concatenation of enriched TFIDF features. Further, the work in Pittaras et al. [2020] combines Wordnet sense statistics and CBOW embeddings [Mikolov et al. 2013] over different lexico-semantic fusion, disambiguation and concept weighting methods. Concept mapping is expanded and regularized with hypernymy-based spreading activation [Collins and Loftus 1975], diffusing semantics in a controlled manner per expansion step. Moreover, document-level taxonomies are built by utilizing hypernymy in Škrlj et al. [2020], leveraging different lexical BoF/TFIDF vector configurations for different terms. Semantic vectors are built via double-normalized TFIDF [Manning et al. 2008] after sense disambiguation, followed by statistical, graph-theoretic and ranking-based feature selection to arrive at a fixed dimensionality.

Additional resources exploited include SentiWordNet for sentiment mining in Kumar and Minz [2013]. The authors extract sentiment scores from mined senses, using TFIDF and various content-based aggregation/selection means. The work in Škrlj et al. [2021] mines ConceptNet to generate semantic relational features, by grounding triplets in the ontology during document traversal and collecting TF-IDF bags of relations for each document. These are subsequently concatenated with different token-level and distributed features for multiple classification tasks.

For images, many works employ relevant ground truth like class hierarchies, label semantics, and localized annotations to create high-level representations with IEM, given a lack of high-level visually-oriented knowledge resources.

For instance, visual annotations to image regions and segments have been exploited as prior semantic knowledge. Such metadata can be utilized for high-level semantic feature generation in classification tasks in a variety of ways, and are usually available as supplementary ground truth information in the Labelset. Early applications [Luo and Savakis 2001] use regions marked as “sky” or “grass” to build intermediate classifiers as semantic feature generators, combined with color and texture features. An array of pre-selected concept annotations to fixed-size regions is used in Vogel and Schiele [2007], fed to classifiers that generate conceptual model vectors as high-level features that are concatenated with multiple low-level features such as color, direction and intensity histograms.

Another useful source of knowledge in image datasets are tags, conveying high-level semantics exploitable for improving performance. Utilization avenues include adopting representation extraction techniques for text, using responses as higher-level semantic features. Given this textual modality data, additional conceptual information can be obtained by probing NLP-oriented knowledge resources such as Wordnet. To this end, the work in [Benitez and Chang 2003] use aggregated color features along with image tags mapped to TF and TFIDF vectors. Multimodal vectors are clustered into knowledge graph of mid-level concepts with various techniques, used as high-level semantics. Wordnet is used for sense disambiguation and facilitation of semantic similarity extraction between graph concepts. Additionally, the authors in Marszalek and Schmid [2007] use textual tags, annotations and labels to extract training examples from Wordnet, exploiting its relational structure (e.g., hypernymy, meronymy, and holonymy) with pruning applied to inhibit propagation towards too generic graph nodes. Sense activations are then combined with SIFT features and a visual BoW approach. Segment predictions are enhanced with tag-based semantics in Kliegr et al. [2008]: first, a named entity recognition (NER) procedure probes Wordnet to discover concepts most similar to an image tagset, via a targeted hypernym discovery process and special measures to improve coverage (e.g., by retrieving relevant Wikipedia articles in terms of textual similarity and article linkage). This procedure results in a collection of senses matching image tags, which are fused with MPEG7 features on regions segmented via self-organizing maps [Kohonen 1990] and particle swarm optimization [Kennedy and Eberhart 1995]. Other approaches include building taxonomy membership features, expressing instances with respect to class memberships in a taxonomic resource. For example, the work in Binder et al. [2009] utilizes the VOC2006 taxonomy [Everingham et al. 2005], exploiting hypernymy-based node membership of the image class in the taxonomy graph to build taxonomic features and affect the training loss by modifying the prediction target, in conjunction with visual features based on SIFT and visual BoW.

Regarding audio data, a similar setting persists, as the visual domain: namely, auxiliary metadata such as ground truth information and textual annotations are the facilitator of enrichment with an IEM strategy.

One such avenue can be found in the music classification domain, where text metadata in the form of song lyrics are exploited for enrichment, via taking advantage of IEM methods for text. Such an approach is adopted by Hu and Downie [2010], utilizing audio and text features: Wordnet, Wordnet-Affect [Strapparava et al. 2004], General Inquirer and E-ANEW provide semantic and psycholinguistic responses, generating statistical, POS and heuristic-based sense and text features in TF and TFIDF ngram bags. These are used to enrich of MFCC, spectral and statistical audio features in various combination configurations. Additionally, in Jamdar et al. [2015] E-ANEW is used to mine valence and arousal scores from song lyrics, with Wordnet powering mapping expansion and semantic disambiguation via sense synonyms and POS information, respectively. Knowledge-based features are adjusted to song-level values and combined with multiple musical, rhythmic and psychoacoustic features, followed by normalization and scaling.

Other approaches exploit relations found in the ground truth and/or labelset of the available audio data. Namely, an early approach uses coarse-grained classification as intermediate high-level features in Zhang and Kuo [1998], with audio-oriented concept scores being used as model vectors, subsequently classified into finer target classes (e.g., rain, bird sounds). Audio content is modeled via signal and spectrum statistics, psychoacoustic/musical features and rule-based clustering. The study in Cano et al. [2004] uses audio samples with Wordnet annotations expanded with additional audio-related concepts. This ground truth is utilized for training a classification system to produce concept-level confidence scores, such as producing assignments to musical instruments. MFCC, statistical, spectral and psychoacoustic features are utilized for the representation, fed to k-NN classifiers for instrument sound categorization.

Here we present a knowledge-guided analog to AGB, modifying features in an informed manner. In KBR, knowledge-based relations can be used as aggregation criteria for similar operations on corresponding content-based features, acting as, e.g., filtering and membership indicators for informed data fusion. Moreover, representation learning methods can inject constraints, regularization, and scaling in the loss/similarity function or fitness objective. For example, taxonomic information linking the concepts “dog” and “wolf” (i.e., subspecies of “Canis”) may be exploited by an embedding generator to bring such data points closer in the embedding space. Likewise, a database of synonyms could make a BoW approach to merge weights matched to concepts “dog” and its synonym “Canis Familiaris”. Such modifications are informed/quantified by utilizing external knowledge of suitable encompassed relationships.

KBR methods for text exploit relations in NLP-related resources to influence representations. One specific approach utilizes word groups, based on categorical assignments, numerical scores or pairwise relations. Intra-group member vectors are then biased to lie close in the embedding space during training or post-processing phases.

A number of works follow this procedure with sentiment-oriented knowledge. For instance, the approach in Li et al. [2017] expands the GloVe model of Pennington et al. [2014] to apply sentiment-guided bias on the word-level, via positive/negative/neutral tags obtained by lexicons with synonym expansion, and globally, via document-level sentiment annotations. Moreover, the work in Yu et al. [2017] introduces refinement of word vectors in a two-stage re-ranking scheme: first, the 10 nearest neighbors of a word are retrieved, followed by re-ranking with respect to the degree of positive and negative sentiment expressed by each. The cosine similarity between word vectors is used to obtain semantic similarity, while the sentiment score is provided by the “valence” norm in the E-ANEW [Warriner et al. 2013] lexicon. The refined output of GloVe and Skipgram vectors then used as the final representation.

Other semantic relationships exploited are paraphrase pairs, i.e., text tuples of different contents but synonymous semantics. Namely, the work in Shi et al. [2019] retrofit contextualized ELMO embeddings [Peters et al. 2018] with paraphrasal information to learn an orthogonal transformation that maps paraphrasal pairs to collocated projections targets. Further, semantic word relationsips of synonymy and antonymy are utilized in works such as in Glavaš and Vulić [2018]. There, the authors seek to optimize with respect to antonymy and synonymy constraints while trying to maintain distances between remaining instance pairs. The investigation explores different optimization objectives, knowledge resources (i.e., Wordnet and Roget’s Thesaurus [Jarmasz and Szpakowicz 2004]) and source embeddings (Skipgram, GloVe, and FastText [Joulin et al. 2017]) to train a mapping with a dense neural network.

Regarding images, one KBR technique exploits metadata with text-oriented enrichment methods, where graph-based resources like Wordnet are useful since they provide multiple exploitable relations via node linkage. For example, early work in Srikanth et al. [2005] uses region descriptions, color, position, texture, shape, and blob-to-word translation associations. They build an “ontology-induced visual vocabulary” by reweighing KMeans with word contributions per cluster, while generative region modeling to Wordnet senses is used to facilitate classification. A similar approach in [Li and Sun 2006] builds a KMeans codebook from keyword-annotated regions, a process augmented with constraints from Wordnet relations that modify the clustering objective to consider the semantic similarity between keywords.

Additional knowledge resources have been exploited to modify representations in the visual domain, such as Imagenet, which provides a visual component to the hierarchy of the Wordnet semantic graph. It is exploited in Deselaers and Ferrari [2011] for computing semantic similarity by extracting the k visually nearest neighbors in Imagenet for each image in an input pair, using a similarity measure based on GIST [Oliva and Torralba 2001], with a final score computed either from pairwise neighbor similarities or the overlap of their category distributions.

Additionally, region-level ground truth and high-level semantic model vectors have been use leveraged in KBR methods. First, image region annotations specifying whole individual objects can instruct representation systems to pool together local descriptors extracted from that region. This is investigated in Wu et al. [2010], where object and foreground annotations bias features extracted from similar regions to be grouped together to same visual words, in a “semantics-preserving” codebook (SPC) process. SIFT with SPC and different distance metric learning techniques such as neighbourhood component analysis (NCA) [Goldberger et al. 2004] are used to build the final representation. Second, a model vector approach may leverage predictions of state-of-the-art classifiers to produce high-level semantics and structures usable for defining meaningful relationships for subsequent representation refinement. Such an approach is pursued by Menglong et al. [2019], using popular CNNs to build a knowledge graph and marking the top k classification results as related, forming pairwise category relationships. Content-based results are refined with a graph-based category similarity measure, improving the accuracy of different deep convolutional nets.

Another refinement avenue approach includes exploiting knowledge-enabled expansion of the available data pool, building the resulting representations with the support of additional data. For instance, in Li et al. [2014], the authors adopt a multi-instance learning approach with “privileged information” [Vapnik and Vashist 2009], i.e., additional extraneous information available during training. This is realized by using retrieval methods to obtain additional related training samples from the web. For training, metadata in the form of image text descriptions are exploited and to TF vectors, while convolutional Decaf features are used for visual content description [Donahue et al. 2014].

Regarding refinement approaches for audio data, existing methods utilize different subdomains of audio-related knowledge in a variety of forms, to produce enrichment via feature refining procedures. For instance, knowledge related to audio signal processing presents one such avenue: this approach is followed in Pachet and Roy [2009], where the “Analytical Features” framework encapsulate knowledge over audio feature engineering and design paradigms (e.g., different practices, heuristics and patterns). These are encoded into operators that process/transform/compose different signal/spectral-based information. The framework is evaluated over fine-grained categorization for artificial and natural sounds (e.g., dog barks, percussion) with a polynomial SVM.

As in other modalities, ground truth presents a rich source of knowledge in audio data. For instance, fine-grained musical annotations can be utilized as conceptual features and high-level representations, which is explored in Cheng et al. [2008] in the form of manually annotated chord transcriptions. These are used to train a chord-ngram HMM model for tasks such as music emotion classification, which captures sequential information of detected chord progressions, along content-based features such as PCP vectors, Longest Common Chord Subsequence similarity, histogram-based measures and MFCC. The work in Zharmagambetov et al. [2022] uses a CNN - LSTM architecture [Hochreiter and Schmidhuber 1997] jointly with a tree-based ontology built from the training labelset. The ontology is used in a decision tree structure to explicitly model the hierarchical probability distributions during learning, which additionally utilizes unlabeled data via consistency training. Further, in He et al. [2021], hierarchical ontologies are manually built by considering dataset and labelset-derived semantics, defining coarse and fine label separations. Ontology levels then determine batch sampling strategies for triplet generation, used to feed a CNN equipped with a triplet loss.

Other types of ground truth include higher level information, such as musical genre, which can be used as conceptual features for generic audio classification. This is investigated in Favory et al. [2020], where an autoencoder scheme is used for aligning content and knowledge in a dual channel architecture. The first channel involves a content-based convolutional autoencoder that ingests spectrogram representations of the audio signal and is trained with a reconstruction loss. The second channel supplies one-hot encodings of semantic audio tags to a feed-forward NN, fitted via cross-entropy. The two learned representations are biased to align via a contrastive loss producing semantically enriched features, fed to an MLP for sound event recognition, instrument and genre classification tasks.

Adhering to the popularity and performance of large end-to-end learning models, KAE approaches utilize knowledge in conjunction with deep learning and neural networks. Here, the injection of external information may occur as input and/or refinement operations, as in previous enrichment strategies—the distinguishing factor is that here, content-based and external information is jointly exploited in an end-to-end fashion, to automatically learn knowledge-enriched feature hierarchies. As in DRL, KAE approaches often jointly learn the representation and discrimination components that facilitate classification, and routinely arrive at highly transferable representations.

A rich collection of knowledge resources have been utilized for KAE methods, leveraging unsupervised training on enriched data inputs for high-performance transfer learning. Entity information is one such knowledge domain, assigning high-level semantics to sequences of words and linking together different entity lexicalizations.

Entity-aware language models for text are investigated in knowBERT [Peters et al. 2019], providing approaches for integrating pretrained models with generic knowledge bases structured as triplets or graphs. Self-attentive mention-span embeddings are computed for each candidate entity, followed by neural entity linking [Kolitsas et al. 2018; Lee et al. 2017]. Lexical embeddings are then re-contextualized with the entity-span vectors, using a multihead attention transformer layer. Wikipedia, YAGO and Wordnet are examined for entity identification, with embeddings built via skipgram vectors. Another approach integrating entities along with phrase semantics is ERNIE [Sun et al. 2019b]. ERNIE expands the BERT model [Devlin et al. 2019] to consider knowledge base information, introduced by knowledge-level masking: phrase-level and conceptual/named-entity-level masking is adopted, extending the byte-pair encoding (BPE) used in BERT. This modeling approach is utilized in pretraining, which is performed on multiple-domains and heterogeneous data. Another model focusing on named entities [Zhang et al. 2019b] uses distinct textual and knowledge encoders to handle lexical/syntactic and fine-grained entity-related information, respectively: in contrast with previous studies, content and knowledge data is fed to the model via different input channels. TransE [Bordes et al. 2013] and Wikidata are used to encode entities into embedding vector, which are combined with lexical token embeddings via multi-headed attention, while TAGME [Ferragina and Scaiella 2010] is used for entity extraction. Entity prediction is added as a pretraining task, utilizing special tokens for entities and masked language modeling.

Furthermore, sentiment and semantic word relationships have been used in KAE; such information can highlight relations between word/sequence pairs, which can be exploited and learned in neural architectures. For instance, in SentiLARE [Ke et al. 2020], POS information is used to extract word-level sentiment polarity from SentiWordnet [Baccianella et al. 2010] to train a multilayer transformer. Masked language modeling is used for pretraining, using context-aware sentiment attention to weigh polarities from individual words to the sentence-level sentiment score. Pretraining subtasks include predicting both sentence and word-level labels. in Chen et al. [2018], lexical relations including synonymy, antonymy and hypernymy are mined from Wordnet in order to enhance premise / hypothesis classification on sentence pairs. The relations are represented as binary vectors, mapped via graph embedding methods (i.e., TransE). The model uses two biLSTM layers for sequence encoding and decoding word relationships, respectively. Semantic lexical information is weighted by the alignment score between word pairs. Moreover, the work in Liu et al. [2022] extracts conceptual information of input words from the Probase taxonomy [Wu et al. 2012], applied in a short text classification setting. Temporal CNN and transformer architectures are used for embedding and representation construction respectively, with the content-based and conceptual channels being merged via cross-attention to enriched features.

Finally, knowledge in the form of dataset metadata is exploited in Li et al. [2022] with the DASK system; it identifies domain-independent words from dataset source information and builds knowledge graphs that encapsulate their relationship to domain-related content. They use a BERT variant on knowledge-injected data to classify user reviews.

Ground truth and labelset information is widely used in KAE methods for images. For instance, the work in Yang et al. [2022] utilizes content-based information with neighborhood embeddings and knowledge from Wordnet via vectorizing textual node descriptions, building a unified knowledge graph. The structure is sampled by graph attention modules for few-shot classification, using separate subgraphs for different tasks for debiasing. Visual Genome has been exploited to construct fitting knowledge structures for task-specific enrichment. A popular such format is visual knowledge graphs, utilized for enriched image classification. For instance, Visual Genome has been used to build a knowledge graph of candidate image labels in Marino et al. [2017], using encapsulated object-object/object-attribute relationships and fusing scene graphs with Wordnet semantics. Classification uses a Graph Search Neural Network [Li et al. 2016], that scores each node in the knowledge graph and provides a global aggregate prediction. An object detector/classifier is used to identify an initial visual node in the image, from which controlled propagation supplies additional neighbor nodes of visual objects. A visual knowledge graph is also built in Zhang et al. [2019a], containing semantic associations between content in the image to objects and scenes relevant to it. Scene labels are produced either by classifiers trained on Imagenet, or semantic associations of the image lexical label, which are extracted from Wordnet. A similarity score based on co-occurrence statistics of objects/scenes detected between images pairs is used to augment learning, fed to different CNNs to predict object/scene labels. Ground truth in the format of scene graphs has also been used in [Li et al. 2019] for visual question answering (VQA), which can be viewed as answer classification over visual and textual inputs. Neural perceptual modules (convolutional, attention-based and dense) are used as specialized operators, each adhering to specific semantic subtasks (e.g., boolean operations, localization of salient/relevant regions, inference) matching the question type structure. Scene graph annotations consisting of object region coordinates, attributes and relations are used to guide the layout generation and optimization, propagating through each module. Further, ground truth from Visual Genome is exploited in Noh et al. [2019] for VQA tasks. First, structured knowledge in the form of visual descriptions and answers/labels from Visual Genome is used to generate blanked image descriptions that characterize visual recognition tasks. To disambiguate candidate tasks, Wordnet sense-based modeling is used: sampling a task specification during training involves retrieving senses with large lexicalization overlaps with the input question. Sampled task-conditional visual classifiers are subsequently used to score candidate answers, using pretrained neural models and attention-based modeling. In Jayathilaka et al. [2021], explicit and inferred pairwise hierarchical label relations (e.g., subsumption, disjointness) are mapped to n-ball conceptual embeddings. They are joined with content-based DCNN features and projected to the conceptual space with an MLP, with the learning process mapping visual features into the conceptual space defined by the ontology embeddings.

Regarding enrichment of audio classification with KAE systems, a variety of knowledge exploitation methods have been tried. As in the visual modality and other enrichment avenues, text-based ground truth has been a popular choice and important knowledge contributor. For instance, in Bertero and Fung [2016], audio data and text transcriptions are used in a multimodal approach, employing MFCC, spectral, signal-based and psychoacoustic features for audio representation. For text, content is mapped to Word2Vec embeddings, bags of ngrams and features related to syntax, sentiment, antonyms, and speaker turn. Wordnet and SentiWordnet are used for extraction of semantics and sentiment polarity, with neural (CNN, RNN) and CRF [Lafferty et al. 2001] components for classification.

Furthermore, labelset-related ground truth has been explored for audio knowledge injection in NNs. For instance, ontology-aware approaches have utilized labelset class relationships—this is investigated in [Jiménez et al. 2018] in multiple ways: first, spectrograms are used with feed-forward NNs to directly model the ontology classes sequentially, i.e., reserving fully-connected layers to produce predictions for each ontology level. Each prediction is subsequently fed to the next level as input features. A second approach uses a siamese NN [Chicco 2021] trained on instance triplets with the Euclidean distance, with intra-class samples being encouraged to map to closely situated vectors in the embedding space. The network includes all potential cases in a two-layer ontology (i.e., matching subclass, only matching superclass, different superclass) and uses the same final classification method as the first architecture. Additional ontology-oriented studies include [Sun and Ghaffarzadegan 2020], where the authors consider labelset relations with a model consisting of a base CNN, followed by an LSTM with feed-forward and graph convolutional networks (GCN) [Zhang et al. 2019] modeling intra and interdependencies between levels of hierarchy in the ontology, respectively. Their system is evaluated in the single and multi-label classification of urban sounds. Additionally, the work in Zhang et al. [2021] utilizes the ASER eventuality knowledge graph [Zhang et al. 2020] to link acoustic event metadata descriptions with rich relations (e.g., conveying causality, temporal, and contingency relations). The generated associations are subsequently exploited via a relation-aware GCN variant for audio event categorization.