[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
Ultrasonic Transducers for In-Service Inspection and Continuous Monitoring in High-Temperature Environments
Next Article in Special Issue
Fusion of Multi-Modal Features to Enhance Dense Video Caption
Previous Article in Journal
An Angular Radial Extended Interaction Amplifier at the W Band
Previous Article in Special Issue
Enhancing Mask Transformer with Auxiliary Convolution Layers for Semantic Segmentation
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 23
Issue 7
10.3390/s23073518
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

by
Jenniffer Carolina Triana-Martinez
1,†,
Julian Gil-González
2,†,
Jose A. Fernandez-Gallego
3,
Andrés Marino Álvarez-Meza
1,* and
Cesar German Castellanos-Dominguez
1
1
Signal Processing and Recognition Group, Universidad Nacional de Colombia, Manizales 170003, Colombia
2
Department of Electronics and Computer Science, Pontificia Universidad Javeriana Cali, Cali 760031, Colombia
3
Programa de Ingeniería Electrónica, Facultad de Ingeniería, Universidad de Ibagué, Ibagué 730001, Colombia
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Sensors 2023, 23(7), 3518; https://doi.org/10.3390/s23073518
Submission received: 24 January 2023 / Revised: 22 February 2023 / Accepted: 16 March 2023 / Published: 28 March 2023
(This article belongs to the Special Issue Deep Learning for Information Fusion and Pattern Recognition)
Browse Figures
Figure 1
<p>GCECDL sketch for multiple-annotator classification holding ground truth prediction and instance-based expert reliability estimation.</p> "> Figure 2
<p>GCECDL-based fully connected architecture for tabular databases. <math display="inline"><semantics> <mrow> <mi>F</mi> <mi>C</mi> </mrow> </semantics></math> stands for a fully connected (dense) layer. <math display="inline"><semantics> <mrow> <mi>F</mi> <mi>C</mi> <mn>1</mn> </mrow> </semantics></math> and <math display="inline"><semantics> <mrow> <mi>F</mi> <mi>C</mi> <mn>2</mn> </mrow> </semantics></math> use a selu activation. A dropout layer (Dropout) is included to avoid overfitting. The MA layer contains two fully connected layers that output the hidden ground truth label and the annotator’s reliability, fixing a softmax and a sigmoid activation, respectively.</p> "> Figure 3
<p>GCECDL-based convolutional architecture for image databases. Four convolutional layers with 3 × 3 patches, 2 × 2 max pooling, and ReLU activations are included. The MA layer outputs the hidden ground truth label and the annotator’s reliability fixing a softmax and a sigmoid activation, respectively.</p> "> Figure 4
<p>Synthetic results—1D dataset. The predictive label’s probability (PLP) is shown, comparing the prediction of our GCECDL (AUC = 0.99) against: DL-CL(MW) (AUC = 0.79), DL-MV (AUC = 0.9), and RCDNN (AUC = 0.99). Label classes are represented as blue, green, and orange.</p> "> Figure 5
<p>GCECDL-based annotators’ performance (reliability) estimation for the synthetic 1D experiment. Orange dashed line depicts (from left to right) the simulated accuracy for each annotator based on Equations (19)–(22). The blue line shows (from left to right) the estimated annotator’s reliability (<math display="inline"><semantics> <msub> <mi>λ</mi> <mi>r</mi> </msub> </semantics></math>).</p> "> Figure 6
<p>Simulated annotators’ for the MNIST database. A 2D t-SNE-based projection of the Mnist training set from the FC1 layer feature maps is shown (some exemplary images are depicted over their 2D projections, where numbers from zero to nine stand for each MNIST class). The ground truth labels are used as a color. The second row depicted the 2D projection of each annotator’s labels in color. The third row presents the confusion matrix of each expert and their achieved accuracy.</p> "> Figure 7
<p>Visual inspection results for the MNIST database. A 2D t-SNE-based projection of the Mnist test set from the FC1 layer feature maps is shown (some exemplary images are depicted over their 2D projections, where numbers from zero to nine stand for each MNIST class). The ground truth labels are shown in color. The second row depicted the 2D projection of the ground truth prediction in color with some explanation maps for visualization purposes. The third row presents the 2D projection of each annotator’s reliability estimation in color.</p> ">
Review Reports Versions Notes

Abstract

:
Supervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. Still, traditional multiple-annotator methods must account for the varying levels of expertise and the noise introduced by unreliable outputs, resulting in decreased performance. In addition, they assume a homogeneous behavior of the labelers across the input feature space, and independence constraints are imposed on outputs. We propose a Generalized Cross-Entropy-based framework using Chained Deep Learning (GCECDL) to code each annotator’s non-stationary patterns regarding the input space while preserving the inter-dependencies among experts through a chained deep learning approach. Experimental results devoted to multiple-annotator classification tasks on several well-known datasets demonstrate that our GCECDL can achieve robust predictive properties, outperforming state-of-the-art algorithms by combining the power of deep learning with a noise-robust loss function to deal with noisy labels. Moreover, network self-regularization is achieved by estimating each labeler’s reliability within the chained approach. Lastly, visual inspection and relevance analysis experiments are conducted to reveal the non-stationary coding of our method. In a nutshell, GCEDL weights reliable labelers as a function of each input sample and achieves suitable discrimination performance with preserved interpretability regarding each annotator’s trustworthiness estimation.

1. Introduction

Conventional Machine Learning (ML) and Deep Learning (DL) techniques utilize a prediction function that maps input data to output targets. In supervised tasks, output values (or “ground truth”) are available for training, but in many real-world scenarios, these values may be unknown or too costly to obtain [1]. With the rise of DL-based approaches, there has been an increasing interest in their use as the primary tool in various classification and regression tasks [2]. However, a crucial factor that dramatically impacts the performance of DL models is the quantity and quality of labeled data used during training [3]. Concerning this, crowdsourcing is a widely recognized approach for obtaining labeled data cost-effectively and efficiently from multiple annotators. It involves the use of online platforms, such as Amazon Mechanical Turk (AMT), to recruit a large number of individuals (annotators) to label the data and provide their subjective interpretation of the unknown ground truth [4].
In the ML field, assigning labels to instances with the help of multiple annotators is a common practice. However, it presents a significant challenge when traditional supervised algorithms are applied because they rely on the assumption that the training labels provided by a single expert are reliable [5]. When multiple annotators with varying levels of expertise are employed, the reliability of the labels becomes uncertain, leading to decreased performance and inaccurate model predictions. Addressing this issue is crucial for developing practical ML and DL models that perform well on real-world datasets.
Traditional supervised learning algorithms cannot account for the varying levels of expertise and the noise introduced by unreliable labels, resulting in decreased performance. Further research is needed to develop methods that can effectively handle the problem of multi-annotator label aggregation and overcome this challenge. Moreover, several issues arise when using DL in a multiple annotator scenario. One of the main challenges is the variability of annotator effectiveness, which may depend on the sample instance presented. Even for a single task, annotators can provide inconsistent or incorrect outputs, leading to noisy labels [6]. These samples can negatively impact the model’s performance by affecting the gradients and making it difficult to converge on a suitable solution [7,8].
Learning from Crowds (LFC) scenarios pose a significant challenge for ML models, and multiple approaches have been developed to tackle this issue. The most commonly used strategy is to adapt supervised learning algorithms and use majority voting for label aggregation. Yet, this approach has limitations since it assumes that annotators have the same level of reliability [9,10]. In addition, incorrect labels and outliers can influence the consensus, decreasing performance. Therefore, more advanced models, such as the Expectation-Maximization (EM) framework, have been considered to address these issues [11,12]. This approach simultaneously estimates true labels and annotator reliability, making more accurate predictions. Another strategy is to train a supervised learning algorithm while also modeling annotator behavior [13]. This approach yields better results than label aggregation and can be used to identify unreliable labelers and remove their outputs from the training set. On the other hand, recent work has shown that relaxing the independence assumption among annotators can lead to more accurate ground truth estimation [14,15]. Then, sophisticated models have been proposed, such as those that use regression tasks to model annotator behavior employing a multivariate Gaussian distribution [16,17]. Moreover, such techniques can help identify the relationships among experts and improve the overall accuracy of the predictions.
Furthermore, learning with noisy labels is a challenging problem in ML and DL. Recently, numerous methods have been proposed for learning noisy labels with DL-based approaches [18]. These methods can involve developing a robust architecture [19,20], enforcing a DL-model with robust regularization techniques [8] and identifying true labeled examples from noisy training data via multi-network [21,22] or multi-round learning [23,24]. In addition, sparse loss functions such as the Mean Absolute Error (MAE) are employed [25,26]. The MAE can help the model focus on correctly labeled examples; however, performance decreases when it is used on large and complex databases [26].
Here, we introduce a Chained Deep Learning (CDL) strategy to learn from multiple noisy annotators in classification tasks. Such an approach allows coding the non-stationary patterns of each annotator regarding the input space while preserving the inter-dependencies among experts. In addition, we combine the capabilities of CDL with a Generalized Cross-Entropy-based loss function aiming to build a model, termed the Generalized Cross-Entropy-based Framework using CDL (GCECDL), that is less prone to outlier annotations. Our proposal is similar to the works in [27,28] in that we use a deep-learning-based strategy to build a supervised learning model in the context of multiple annotators. Moreover, network self-regularization is accomplished by predicting each labeler’s reliability within our chained scheme. On the other hand, the proposed research uses t-distributed Stochastic Neighbor Embedding- (t-SNE) [29] and Gradient-based Class Activation Maps [30] to interpret and validate the obtained results visually. Finally, experimental results related to multiple-annotator classification on several well-known datasets (synthetic and real-world scenarios) demonstrate that GCECDL outperforms state-of-the-art techniques.
The agenda for this paper is as follows: Section 2 summarizes the related work. Section 3 describes the methods. Section 4 and Section 5 present the experiments and discuss the results. Finally, Section 6 outlines the conclusions and future work.

2. Literature Review

Several approaches have been developed to address LFC scenarios. In this light, we recognize two main groups: combining the annotations to estimate the gold standard or adapting supervised learning algorithms to that type of labels [31]. The primary method is called label aggregation. One of the most used techniques is known as Majority Voting (MV), which has been applied to different multi-labeler problems due to its simplicity [32]. In MV, the most frequent output among the experts is chosen as the final prediction. The latter is simple to implement and effective in some cases, but it also has limitations because MV relies on the assumption that annotators have the same level of reliability, which is challenging to fulfill in real-world scenarios. Additionally, the consensus can be heavily influenced by incorrect labels and outliers [10]. In this sense, EM methods have been proposed to handle imbalanced labeling to handle these issues [32,33]. The EM framework aims to estimate the true label and the annotator’s reliability simultaneously, while methods to handle imbalanced labeling try to adjust for the differences in the annotator’s expertise level. These more advanced models provide more robust solutions to the problem of multi-annotator label aggregation and can lead to better performance than MV.
An alternate approach to addressing multi-labeler tasks is to simultaneously train a supervised learning algorithm while also modeling the behavior of the annotators. It has been empirically demonstrated that this approach yields better results than label aggregation. Furthermore, features that train the learning algorithm can also be exploited to infer the ground truth [13]. One introductory study in this field is the EM-based framework presented by authors in [34], which estimates the sensitivity and specificity of annotators while also training a logistic regression classifier. This approach has served as a foundation for various algorithms that address multi-labeler tasks, including regression [35,36], binary classification [37,38], multi-class classification [4,39], and sequence labeling [40]. Likewise, some studies have adapted these ideas for DL techniques by incorporating an additional layer that contemplates multiple labelers [27,28].
In turn, the study in [15] represents an early exploration of the relationship between annotators’ parameters and input features. The authors propose a method for binary classification with multiple labelers, where input data are grouped using a Gaussian Mixture Model (GMM). The algorithm posits that annotators have specific performance levels in terms of sensitivity and specificity. However, it does not incorporate information from multiple experts as input for GMM, which may result in labelers’ parameter deviation. Likewise, the authors in [6] presented a binary classification algorithm that employs Bernoulli and Gaussian distributions to code the annotators’ performance as a function of the input space. In addition, a linear relationship between the annotator’s expertise and the input space is assumed, which can be problematic. For example, when assessing documents online, annotators may have varying levels of labeling accuracy. These differences could be due to their familiarity with specific topics related to the studied documents [41]. Moreover, in [36], a Gaussian Process (GP)-based regression procedure is proposed to incorporate multiple annotators. The annotators’ parameters are estimated as a nonlinear function of the input space by using an additional GP. Nevertheless, since this approach is based on a classical formulation of GPs, its computational complexity is prohibited for large datasets [39]. Furthermore, relaxing the independence assumption among annotators has led to a more accurate estimation of the ground truth, as demonstrated in [14,15]. In [17], an unsupervised regression task is described where the labelers’ behavior is modeled using a multivariate Gaussian distribution, with the covariance matrix encoding the annotators’ interdependencies. Again, the authors in [38] proposed a binary classification method that utilizes a weighted combination of classifiers. The weights are estimated using a kernel alignment-based algorithm.
Of note, when multiple annotators are present, the training set may contain noisy labels, negatively impacting the model’s generalization ability. Typical regularization approaches, such as dropout and batch normalization, only partially outweigh the over-fitting drawback for DL [18]. An alternative is to use more robust loss functions to train DL models. Because of its fast convergence and generalization capability, most deep learning-based classifiers use Categorical Cross-Entropy (CE) as cost function. Nevertheless, MAE has been found to perform better when dealing with noisy labels [26]. However, the robustness of MAE can concurrently cause increased difficulty in training and lead to a performance drop. This limitation is particularly evident when using neural networks on complicated datasets. To combat this drawback, Zhang et al. [42] proposed the GCE, establishing a more general type of noise-robust loss function taking advantage of both MAE and CE, yielding good performance in the presence of noisy labels. Moreover, it can be readily applied to any existing neural network architecture.
Our proposal follows the lines of the work in [27,28] in that GCECDL uses a deep-based approach to build a supervised learning model in the context of multiple-annotator classification. Yet, while such approaches code the annotators’ parameters as fixed points, we model them as functions to consider dependencies between the input features and the labelers’ behavior. GCECDL is also similar to the works in [14,43]. Both approaches model the annotators’ performance as a function of the input instances and consider the interdependencies among the labelers. Even so, unlike [14], where it is necessary to use as many classifiers as annotators, our approach only needs to train a single classifier from a DL representation, which is advantageous for a large number of labelers. Moreover, unlike [43], our loss function can deal with noisy labels and more difficult training scenarios by using a generalization between MAE and CE. Indeed, network self-regularization is accomplished by predicting each labeler’s reliability.

3. Methods

3.1. Generalized Cross-Entropy

Let us consider a K-class classification problem from a given prediction function f : R P → [ 0 , 1 ] K , trained on the input–output set { X ∈ R N × P , Y ∈ { 0 , 1 } N × K } and holding P — dimensional input features in N row vectors x n ∈ R P corresponding to each ground truth label y n ∈ { 0 , 1 } K , n ∈ { 1 , 2 , … , N } . The prediction function f ( · ) is commonly coupled by a softmax output to fulfill 1 − K one-hot labels. In turn, the well-known Mean Absolute Error (MAE) and Categorical Cross-Entropy (CE) losses, used typically for optimizing f, are defined as follows:
M A E ( y , f ( x ) ) = ∥ y − f ( x ) ∥ 1 ,
C E ( y , f ( x ) ) = ∑ k = 1 K y k log ( f k ( x ) ) ;
where y k ∈ y , f k ( x ) ∈ f ( x ) , and ∥ · ∥ 1 stands for the l 1 -norm. Of note, 1 ⊤ y = 1 ⊤ f ( x ) = 1 , 1 ∈ { 1 } K being an all-ones vector. In addition, the MAE loss can be rewritten for softmax outputs, yielding:
M A E ( y , f ( x ) ) = 2 ( 1 − 1 ⊤ ( y ⊙ f ( x ) ) )
where ⊙ stands for the Hadamard product.
On the one hand, CE is sensitive to label noise, being a nonsymmetric and unbounded loss function. On the other hand, MAE is noise-robust because of its symmetric property, that is [26]:
∑ k = 1 K M A E ( y , f ( x ) | k ) = ∑ k = 1 K 2 ( 1 − f k ( x ) ) = C , ∀ x ∈ R P , ∀ f ;
where C ∈ R + . The symmetric property of MAE for softmax-based outputs allows extending the L1-norm expression in Equation (1) to a vectorized form in Equation (3). Note that the symmetric property is only fulfilled for softmax-based representations. Therefore, the L1-norm favors sparse coding when computing the mismatch between target and prediction, favoring the filtering of noisy outputs, as commonly studied for L1-based filtering approaches [44].
Though MAE is symmetric and bounded, it also has some drawbacks when used as classification loss for deep learning networks trained on large datasets employing stochastic gradient-based techniques. Specifically, for a given network with parameter set θ , the MAE and CE gradients can be computed as:
∂ M A E ( y , f ( x ; θ ) | k ) ∂ θ = − ∇ θ f k ( x ; θ ) ,
∂ C E ( y , f ( x ; θ ) | k ) ∂ θ = − 1 f k ( x ; θ ) ∇ θ f k ( x ; θ ) .
As seen in Equations (5) and (6), less congruent samples have greater weights in CE than predictions that agree more with ground truth labels; meanwhile, the MAE penalizes equally during gradient descent optimization. At first glance, MAE can deal with noisy labels; still, this can lead to longer and more difficult training scenarios, particularly for large databases.
Therefore, authors in [42] proposed a trade-off between MAE and CE using a Box–Cox transformation [45], yielding to the following Generalized Cross-Entropy (GCE) loss for training deep learning models:
G C E ( y , f ( x ) ) = 2 1 − 1 ⊤ ( y ⊙ f ( x ) ) q q ,
with q ∈ ( 0 , 1 ] . Remarkably, the limiting case for q → 0 in GCE is equivalent to the CE expression, and when q = 1 , it equals the MAE loss. In addition, the GCE in Equation (7) holds the following gradient with regard to θ :
∂ G C E ( y , f ( x ; θ ) | k ) ∂ θ = − f k ( x ; θ ) q − 1 ∇ θ f k ( x ; θ ) .
As depicted in Equation (8), the GCE’s gradient weighs samples using the f k ( x ; θ ) q − 1 factor, which could affect robustness against noisy labels depending on the hyperparameter value q. In summary, the larger the q value, the more noise robustness is attained. Therefore, a suitable q is required to find a trade-off between noisy robustness and better learning dynamics during network training.
Lastly, since tighter loss bounding would imply more robust noise tolerance, GCE can be extended to its truncated version as follows [42]:
T G C E ( y , f ( x ) ; λ ˜ x , C ˜ ) = λ ˜ x 1 − 1 ⊤ ( y ⊙ f ( x ) ) q q + ( 1 − λ ˜ x ) 1 − ( C ˜ ) q q ,
where λ x ∈ [0,1] and C ˜ ∈ (0, 1). Note that λ x prunes samples regarding a noise tolerance ruled by C ˜ .

3.2. Chained Deep Learning Fundamentals

The seminal Chained Gaussian Processes (CGP) in [46] fixes a likelihood function with J parameters depending on the input–output set { X , Y } , as follows:
p ( Y | X , ξ ) = ∏ n = 1 N p ( y n | ξ 1 ( x n ) , … , ξ J ( x n ) ) ,
ξ = ξ 1 … , ξ J ⊤ ∈ R N J being a parameter vector and ξ j = ξ j ( x 1 ) … ξ j ( x N ) ⊤ ∈ R N . In addition, each ξ j ( x ) ∈ M j maps an input instance to the parameter space ( j ∈ 1 , 2 , … , J ). The likelihood in Equation (10) allows modeling the function parameters with J independent GPs (one GP prior per parameter).
Likewise, we can extend the concept of CGP to the field of DL. Hence, suppose a DNN with L layers, where the output layer contains J outputs (neurons). The DNN model can be represented by the following composite function f ( x ) = [ f 1 ( x ) , … , f J ( x ) ] ⊤ ∈ R J ,
f ( x ) = ( φ L ∘ φ L − 1 ∘ ⋯ ∘ φ 1 ) ( x ) ,
where ∘ stands for function composition. Accordingly, Chained Deep Learning (CDL) links each likelihood parameter ξ j ( x ) to one of the J outputs. Each CDL parameter can be estimated as: ξ j ( x ) = h j ( f j ( x ) ) , where h j : R → M j maps each f j ( x ) prediction to M j . Furthermore, each function φ l ( · ) , with l ∈ 1 , 2 , … , L , depends on a set of parameters ϕ = [ ϕ 1 , … , ϕ L ] ⊤ , e.g., weights and biases, that can be optimized via gradient descent and back-propagation [47].

3.3. Generalized Cross-Entropy-Based Chained Deep Learning for Multiple Annotators

Nowadays, in several real-world classification problems, instead of the ground truth Y , multiple labels are provided by R experts with different levels of ability, e.g., Multiple Annotators (MA) [28]. For the sake of clarity, we assume that the r-th expert annotates | Ω r | ≤ N instances, | Ω r | being the cardinality of the set Ω r containing the indices of samples labeled by annotator r. Moreover, let Ψ n be the index set gathering the annotators who labeled the n-th instance. Then, a multiple annotators dataset { X ∈ R N × P , Y ˜ ∈ { 0 , 1 , ⌀ } N × K × R } , where y ˜ n r ∈ { 0 , 1 , ⌀ } K is the 1 − K one-hot label of expert r for instance n, can be built to feed a CDL approach holding J = R + K outputs. The former R outputs code each expert reliability λ n r ∈ 0 , 1 , and the remaining K predictions approximate the ground truth y n .
In this sense, given an input sample x , each annotator’s reliability can be predicted by fixing a sigmoid activation to the first R neurons within layer L in Equation (11), as:
λ ^ n r = 1 1 + exp ( − f r ( x n ) ) ,
where λ ^ n r ∈ [ 0 , 1 ] .
Moreover, a softmax function is set to the last K outputs in ϕ L ( · ) to predict the ground truth label, as follows:
y ^ k = exp ( f R + k ( x ) ) ∑ i = R + 1 J exp ( f i ( x ) ) .
where k = { 1 , 2 , … , K } , y ^ k ∈ [ 0 , 1 ] , and ∑ k y ^ k = 1 .
Here, to circumvent noisy annotators while coding their non-stationary behavior along the input space, and to favor the CDL training ruled by the optimization of the parameter set ϕ , a TGCE-based loss as in Equation (9) is proposed for multiple-annotator classification, yielding:
ϕ ∗ = arg min ϕ ∑ n = 1 N ∑ r ∈ Ψ n λ ^ n r ( ϕ ) ∑ k = 1 K y ˜ n , k r 1 − y ^ n , k q ( ϕ ) q + ( 1 − λ ^ n r ( ϕ ) ) 1 − ( 1 K ) q q
where y ˜ n , k r ∈ y ˜ n r and q ∈ ( 0 , 1 ] .
As seen above, self-regularization is achieved through each expert’s reliability estimation λ ^ n r ( ϕ ) in Equation (14), which prunes the TGCE loss ruled by q. Of note, when λ ^ n r ( ϕ ) → 1 , y ^ n ( ϕ ) ∈ [ 0 , 1 ] K holds the 1-K ground truth predictions as in Equation (13). As a consequence, only samples with λ ^ n r ( ϕ ) → 1 are kept for updating the CDL parameters. In contrast, noisy or unreliable annotations ( λ ^ n r ( ϕ ) → 0 ) are avoided to update the network parameters. Therefore, our GCE-based CDL, termed GCECDL, allows coding the non-stationary patterns of each annotator regarding each input instance space while preserving the interdependencies among experts through a CDL approach. In summary, our approach benefits CDL and GCE by circumventing noisy experts with non-stationary patterns. Figure 1 summarizes the GCECDL sketch.

4. Experimental Set-Up

The following section comprehensively describes the tested datasets and the key experimental conditions utilized.

4.1. Tested Datasets

Our GCECDL approach, designed for multiple-annotator classification, is evaluated using synthetic and real-world datasets. The experiments aim to uncover the key insights and advantages of GCECDL for coding non-stationary and unreliable expert labels on complex datasets.
We generate synthetic data for a 1-dimensional, 3-class classification problem by randomly sampling 5000 points from a uniform distribution within the interval [ 0 , 1 ] and using these points to construct the input feature matrix X . The true label y n , k for each sample is determined by taking the maximum value of t n , k for k in the set 1, 2, 3, where t n , 1 = sin ( 2 π x n ) , t n , 2 = − sin ( 2 π x n ) , and t n , 3 = − sin ( 2 π ( x n + 0.25 ) ) + 0.5 . We also create a test set by extracting 2000 equally spaced samples from the same interval.
We then look for datasets where the input data come from real-world applications. Still, the labels from multiple annotators are obtained synthetically. The synthetic labeling is carried out to control the labeling process. In particular, six binary and multi-class classification task datasets are studied from the famous UCI repository (http://archive.ics.uci.edu/ml, accessed on 19 August 2022). The chosen datasets include: Occupancy Detection (Occupancy), Skin Segmentation (Skin), Tic-Tac-Toe Endgame (tic-tac-toe), Iris Plants (Iris), Wine (Wine), and Image Segmentation (Segmentation). Moreover, the Fashion-MNIST dataset [48], as well as the Balance and New Thyroid datasets, are also selected from the Keel-dataset Repository (https://sci2s.ugr.es/keel/category.php?cat=clas, accessed on 3 October 2022). In addition, the publicly available bearing data collected by the Case Western Reserve University (Western) are used. The aim is to build a system to diagnose an electric motor’s status based on two accelerometers. The feature extraction is performed as in [49]. We also evaluate our proposed GCECDL classifier on two large image classification sets: MNIST of Handwritten Digits (MNIST) [50], an easily interpretable image database of labeled handwritten digits with 60,000 images for training and 100,000 for test sets, and the Cats vs. Dogs database, consisting of 25,000 images of dogs and cats [51], each class being represented by 1 and 0, respectively.
Finally, we include three real-world datasets provided with human annotations. First, the CIFAR-10H comprises over 500 k crowdsourced human categorization judgments obtained through AMT and includes ten categories: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck [52]. In our study, we applied a rigorous data-filtering process and discarded any samples that at least one annotator did not label. This filtering step resulted in 19.233 labeled samples, out of which 10.000 were used for testing. The second dataset is LabelMe, which aims to classify images into eight different classes: highway, inside city, tall building, street, forest, coast, mountain, and open country. It consists of 2688 images; each image was labeled by an average of 2547 workers, with a mean accuracy of 69.2%. We used the prepared dataset from [28], which performs a feature extraction stage based on a pre-trained VGG-16 deep neural network [53]. The third one is the Music genre database [54], comprising one thousand 30-second samples of songs categorized into classical, country, disco, hip-hop, jazz, rock, blues, reggae, pop, and metal. Each class contains 100 representative samples. A random selection of 700 samples was published on the AMT platform for workers to classify them from one to ten based on their genre. Feature extraction was performed following the method outlined in [55], resulting in an input space of 124 features. Table 1 summarizes the tested synthetic, semi-synthetic, and real-world datasets.

4.2. Provided and Simulated Annotations

To test our GCECDL classifier, we simulate annotator labels as corrupted versions of the hidden ground truth. Here, the simulations are performed by assuming: (i) dependencies among annotators and (ii) the labelers’ performances are modeled as a function of the input features. In turn, the semiparametric latent factor model is used to build the labels, as follows [56]:
  • Define Q deterministic functions μ ^ q : χ → R and their combination parameters ω ^ l r , q ∈ R , ∀ r ∈ R , n ∈ N .
  • Compute f ^ l r , n = ∑ q = 1 Q ω ^ l r , q μ ^ q ( x ^ n ) , where x ^ n ∈ R is the n-th component of x ^ ∈ R N , x ^ being the 1-D representation of the input features in X by using t-SNE approach [29].
  • Calculate λ n r = sigmoid ( f ^ l r , n ) , where sigmoid ( · ) ∈ [ 0 , 1 ] is the sigmoid function.
  • Finally, find the r-th label as y r n = y n if λ r n ≥ 0.5 y ˘ n if λ r n < 0.5 , where y ˘ n is a flipped version of the actual label y n .

4.3. Performance Measures, Method Comparison, and Training

As quantitative assessment concerning the classification performance, the overall Accuracy (ACC) and the Balanced Accuracy (BACC) are reported on the testing set, which can be written as follows:
A C C [ % ] = 100 T P + T N T P + T N + F P + F N
B A C C [ % ] = 100 2 T P T P + F N + T N T N + F P
where T P , F N , and F P represent the true positive, false negative, and false positive predictions, respectively, after comparing the actual and estimated labels y n and y ^ n for a given input sample x n .
In addition, we consider the Normalized Mutual Information (NMI) between the output and the target [57]. The NMI measures the amount of shared information between two variables and quantifies the strength of their relationship, yielding:
NMI [ % ] = 100 1 N ∑ n = 1 N 2 I ( y n , y ^ n ) H ( y n ) + H ( y ^ n ) ,
where I ( · , · ) stands for mutual information and H ( · ) for marginal entropy. Furthermore, we estimate the Area Under the ROC Curve (AUC) that can be computed by varying the decision boundary concerning the sensitivity (Sen) and specificity (Spe) measures, as follows [47]:
A U C [ % ] = 100 S e n + S p e 2
For concrete testing, we use a cross-validation scheme with 10 repetitions holding 70% of the samples for training and the remaining 30% for testing (except for the Mnist, F-Mnist, CIFAR-10H, and music dataset, where training and testing sets are clearly defined).
Moreover, Table 2 displays the tested state-of-the-art algorithms for comparison purposes. The abbreviations are fixed as follows: Regularized Chained Deep Neural Network Classifier for Multiple Annotators (RCDNN) [43], Deep Learning Majority Voting (DL-MV), and Deep Learning from Crowds (DL-CW(MW)) [28]. Our Python codes are publicly available for DL-CL(MW), DL-MV, RCDNN, and GCECDL at https://github.com/Jectrianama/GCCE_TEST (accessed on 19 December 2022). Regarding DL-CL(MW), we use the codes at http://www.fprodrigues.com/ (accessed on 19 December 2022). Of note, DL-GOLD is a deep learning model trained with the true label, which is used only to provide an upper bound.
In turn, to better grasp the behavior of our GCECDL classifier over every dataset, we implemented a grid-search scheme to fix the q value within the grid [ 0.001 , 0.1 , 0.2 , 0.3 , 0.4 , 0.5 , 0.75 ] .
The proposed GCECDL architecture for multiple annotators comprises (i) a fully connected network for tabular data (see Figure 2) and (ii) a convolutional network for image data (see Figure 3). For all provided layers, elastic-net-based weight regularizers are used. As usual, the optimization problem is solved using a back-propagation algorithm. Moreover, to favor scalability, we utilize a mini-batch-based gradient descent approach with automatic differentiation (the Adam-based optimizer is fixed). In addition, we employed callbacks during the training process to monitor the model’s performance. Specifically, we used an EarlyStopping callback to stop the training process if the validation performance did not improve for a specified number of epochs and a LearningRateScheduler callback, allowing the model to converge more quickly by avoiding becoming stuck in a suboptimal solution. These callbacks allowed us to optimize the performance of our neural network and sidestep overfitting. Finally, we selected the best performance between models with or without callbacks for each database. All experiments were conducted in Python 3.8, with the Tensorflow 2.4.1 API, on a Google Colaboratory environment.

5. Results and Discussion

5.1. Reliability Estimation and Visual Inspection Results

We first perform a controlled experiment to test the GCECDL’s capability when dealing with binary and multiclass classification. We use the one-dimensional synthetic dataset described in Section 4. In addition, five labelers (R = 5) are simulated with different levels of expertise. To simulate the error variances, we define Q = 3 μ ^ q ( · ) functions (see Section 4.2), yielding:
μ ^ 1 ( x ) = 4.5 cos ( 2 π x + 1.5 π ) − 3 sin ( 4.3 π x + 0.3 π )
μ ^ 2 ( x ) = 4.5 cos ( 1.5 π x + 0.5 π ) + 5 sin ( 3 π x + 1.5 π )
μ ^ 3 ( x ) = 1 ,
where x ∈ [ 0 , 1 ] . In addition, the combination weights are gathered within the following combination matrix W ^ ∈ R Q × R :
W ^ = 0.4 0.7 − 0.5 0.0 − 0.7 0.4 − 1.0 − 0.1 − 0.8 1.0 3.1 − 1.8 − 0.6 − 1.2 1.0
holding elements ω ^ l r , q , which are used to combine functions μ ^ 1 , μ ^ 2 , and μ ^ 3 .
For visual inspection purposes, Figure 4 shows the predictive label’s probability (PLP) and the AUC, for all studied approaches regarding the one-dimensional synthetic database.
As seen in Figure 4, DL-CL(MW) and RCDNN have a different shape than the ground truth. Additionally, DL-MV has the worst accuracy for two out of the three classes. Upon further analysis of the results of our GCECDL method, we note that its predictive accuracy is quite close to the absolute ground truth, which is the theoretical upper limit. Thus, GCECDL offers a more suitable representation of the labelers’ behavior compared to its competitors. This is because GCECDL takes into account both the annotators’ dependencies and the relationship between the input features and the annotators’ performance.
To support the previous statement, Figure 5 illustrates the per-annotator reliability estimated by our model and the simulated accuracy of each annotator. As can be seen, our model provides an excellent representation for annotators one and five and an acceptable representation for annotators two, three, and four. This is a direct result of modeling the labelers’ parameters as functions of the input features. This outcome demonstrates how our approach effectively identifies the areas where a specific labeler aligns with the regions of higher accuracy of the simulated annotators.
In this next step, we conduct two crucial experiments utilizing the MNIST dataset, as outlined in Section 4. These experiments include an examination of explainable multiple-annotator classification and a t-SNE-based 2D visualization of the data. To achieve this, we employed the Gradient-weighted Class Activation Mapping (Grad-CAM++) approach to extract normalized class activation mapping from the image data, as described in [30]. We then plotted heatmap images related to the FC 2 layer, which represents the high-level visual features that are extracted from the characteristics. These feature maps are then projected onto a two-dimensional space using the t-SNE algorithm. The visual analysis of these results shows that the same color image samples cluster together in the 2D, low-dimensional space according to their class, while preserving the spatial relationships from the input space.
Figure 6 presents a visualization of the gold standard and simulated annotators ( R = 3 for illustrative purposes) plotted over the resulting 2D features projection for the training set. It can be observed that the features extracted possess a high degree of separability and discriminative ability, as every class (0–9) is represented by a distinct cluster. For illustration purposes, a few images are depicted over each corresponding projection. The last two rows show a selection of simulated labels and their different scores (annotator reliability). We can see the different levels of expertise obtained from the confusion matrix. The first annotator, whose accuracy score with respect to the ground truth labels is 97%, is depicted over the projection. We can observe how most samples have a correct version of the ground truth. However, it tends to fail more for classes 0, 2, 3, and 8. This behavior becomes more pronounced for the last two annotators, whose accuracy drops to 41% and 11%, respectively. The mismatch between the labels and the ground truth is more evident in the top figure. We expected to compare this with the estimated reliability obtained by our model.
Then, Figure 7 illustrates the hidden ground truth prediction and reliability estimation generated by our GCECDL approach. As shown, GCECDL demonstrates a high level of suitability for the MNIST digit classification problem by achieving an ACC score of 0.99, which highlights its generalization capability, even in cases where the ground truth is unknown. This is because the proposed model takes into account both the relationship between the input space and the annotator’s behavior, as well as the dependencies among their labels, which improves the quality of expert codification, as described in [4,38]. To provide further insights, we also generated visual explanations for a subset of the samples in the test dataset. To achieve this, Grad-CAM++ was applied to a given image and class K to determine the regions of the image that are most competitive for classification. As seen in the CAM, the important regions highlighted in red reveal that our model can effectively exploit the most relevant features to correctly identify the image’s class. The second row of the figure depicts some visual explanations on the 2D projections. Notably, class 4 can be confused by the model with a seven or a nine in a few samples.
In the last row of Figure 7, we can infer that our method effectively identifies the zones where the labelers have the best accuracy. This is not unexpected as the annotators’ accuracy (simulated) is compared with their reliability (estimated); therefore, the clusters where a specific labeler obtains the highest accuracy should align with the clusters where the estimated reliability is closest to 1 (yellow). To further support this statement, we depict the estimated probability function through a kernel density estimation (KDE) plot, to show the reliability estimated per annotator. For example, regarding annotator 1 (blue), as most of its estimates are reliable, the KDE increases when the reliability is 1. However, for annotator two (orange), its peak KDE value is slightly lower when the reliability is one. Similarly, annotators 3 (green) and 4 (red) exhibit an inverse behavior, as their performance is more doubtful.
In addition, it is important to note that our proposed GCECDL encodes the interdependence between annotators. By comparing the simulated annotators and the labelers’ performance in Figure 6 and Figure 7, it is clear that our proposed model closely follows the performance pattern of the labelers’ capabilities. Therefore, when a suitable labeler is presented, the model provides a high estimation for the same labeler in the zones where the labelers have the best accuracy. Conversely, when malicious labelers are present, the model reflects this in poor reliability estimation. This highlights how the ability to label and reliability estimation are closely related. Furthermore, it is worth noting that annotators with high uncertainty tend to have CAMs with more energy, which supports the aforementioned statement empirically.

5.2. Method Comparison Results for Multiple-Annotator Classification

Table 3 presents the results of our experiments on real-world datasets. The non-parametric Friedman test results for every quantitative assessment measure establish their statistical significance. The null hypothesis is rejected, indicating that all algorithms attain different performances as described in [58]. Additionally, the significance threshold is fixed at p < 0.05 . The DL-GOLD standard is not included in the test to compare state-of-the-art approaches exclusively. It is worth noting that most classification schemes exhibit a highly satisfactory performance for the datasets with simulated annotators, as reflected in the scores.
Our approach outperforms the selected state-of-the-art methods across most evaluation measures. For example, our proposal achieves the highest accuracy on 12 of 15 datasets, as shown in Table 3. Similarly, GCECDL outperforms tested strategies regarding BACC, NMI, and AUC for most datasets. This outcome is due to the fact that GCECDL properly codes annotators’ reliability by considering the correlations among their decision, even for noisy outputs. Remarkably, RCDNN reaches the second highest average performance. Indeed, it obtains the highest performance on CIFAR-10H, LabelMe, and Music. In general, RCDNN is similar to GCECDL, but it is significantly affected when the annotators’ performance is below 60%. Furthermore, it struggles with databases with a high class imbalance ratio. This behavior highlights that RCDNN is more susceptible to noisy or imbalanced scenarios.
Moreover, we observe that high NMI values for some datasets suggest that the labels provided by the annotators are consistent and reliable. Our GCECDL method effectively captured these dependencies. For example, in the MNIST dataset, a high NMI value of 97.39 % is attained, indicating that the labels provided by the different annotators and the hidden ground truth prediction are highly consistent. In contrast, in the Cats vs. Dogs dataset, we observe a lower NMI value of 20.79 % , pointing to more significant variability in the annotations and highlighting the challenges of dealing with noisy or inconsistent annotations.
Finally, the DL-CL(MW) approach outperforms the DL-MV scheme, which can be observed in the average ranking. Furthermore, it is worth noting that DL-CL(MW) includes the introduction of the CrowdLayer, which allows for training neural networks directly from multiple labels without encoding the annotators’ behavior. On the other hand, DL-MV presents the lowest performance among the studied methods. It can be explained by the fact that DL-MV is the most naive approach, and most annotators were simulated with a low level of expertise, negatively impacting the outcome of the majority voting strategy.

6. Conclusions

This article introduced a Generalized Cross-Entropy-based Chained Deep Learning model, termed GCECDL, to deal with multiple-annotator scenarios. Our method follows the ideas of [43,46], where each parameter is modeled in a multi-labeler likelihood by using the outputs of a deep neural network. Nonetheless, unlike [43]—where a CCE-based loss was used—we also introduced a noise-robust loss function based on GCE [42] as a tradeoff between MAE and CCE. Thus, GCECDL codes the non-stationary patterns of each annotator regarding the input space. We tested our approach for classification tasks using fully synthetic and real-world databases from well-known repositories, including structured data and images. According to the results, our GCECDL can achieve robust predictive properties for the used datasets defeating the selected state-of-the-art models. We attribute this behavior to the coupled MAE and CE within GCE, exploiting the symmetry property of MAE for softmax-based outputs and the L1-norm and cross-entropy tradeoff in weighting noisy annotations as a function of the input space. In addition, our chained architecture yields a self-regularization strategy within a DL framework that favors proper labeler’s reliability estimation and ground truth prediction. On the other hand, we created visual explanations using GradCam++ [30] to identify the most influential regions that demonstrate the model’s ability to correctly predict the hidden ground truth and assess the reliability of the annotators. Furthermore, using t-SNE [29] to project the extracted features onto a two-dimensional space allowed us to retain the spatial relationships from the original input space.
As future work, extending the GCECDL for regression tasks is a promising research area, as demonstrated by the model introduced in [35]. Our next step is to experiment with various activation functions and deeper convolutional and recurrent architectures to tackle complex tasks such as computer vision, natural language processing, and graph-based modeling [59]. Additionally, we plan to develop a model for identifying non-stationary patterns from multidomain input spaces holding noisy targets in an agricultural context, using multispectral imagery, climatic data, and infield data, instead of relying on annotators [60]. Furthermore, we plan to test more Explainable AI methods to provide deeper insight into our model performance [61,62,63], e.g., Layer-wise Relevance Propagation that captures both negative and positive relevance. Finally, actionable and explainable AI extensions based on our GCECDL would be an exciting research line [64].

Author Contributions

Conceptualization, J.C.T.-M., J.G.-G. and A.M.Á.-M.; data curation, J.C.T.-M.; methodology, J.C.T.-M., J.G.-G. and A.M.Á.-M.; project administration, C.G.C.-D.; supervision, C.G.C.-D., J.A.F.-G. and A.M.Á.-M.; resources, J.C.T.-M., J.G.-G. and J.A.F.-G. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded (APC) by the project “Herramienta de apoyo a la predicción de los efectos de anestésicos locales vía neuroaxial epidural a partir de termografía por infrarrojo” (Code 111984468021-Minciencias). Also, G. Castellanos thanks to the project “Desarrollo de una herramienta de visión por computador para el análisis de plantas orientado al fortalecimiento de la seguridad alimentaria” (HERMES 54339-Universidad Nacional de Colombia and Universidad de Caldas). J. Triana thanks to the program “Beca de Excelencia Doctoral del Bicentenario-2019-Minciencias” and the project “Rice remote monitoring: climate change resilience and agronomical management practices for regional adaptation—RiceClimaRemote” (Flanders Research Institute for agriculture, fisheries and food ILVO-Government of Flanders-Belgium, Universidad de Ibagué, and Agrosavia).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Publicly available datasets were analyzed in this study. These data can be found at the UCI repository (http://archive.ics.uci.edu/ml, (accessed on 19 August 2022)), Keel-dataset repository (https://sci2s.ugr.es/keel/category.php?cat=clas, (accessed on 3 October 2022)), and Rodrigues repository (http://www.fprodrigues.com/, (accessed on 20 Sepetember 2022)). Fashion-Mnist is freely available at dataset repository (https://github.com/zalandoresearch/fashion-mnist, (accessed on 27 August 2022)).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Zhang, J.; Sheng, V.S.; Wu, J. Crowdsourced label aggregation using bilayer collaborative clustering. IEEE Trans. Neural Netw. Learn. Syst. 2019, 30, 3172–3185. [Google Scholar] [CrossRef] [PubMed]
  2. Parvat, A.; Chavan, J.; Kadam, S.; Dev, S.; Pathak, V. A survey of deep-learning frameworks. In Proceedings of the 2017 International Conference on Inventive Systems and Control (ICISC), Coimbatore, India, 19–20 January 2017; pp. 1–7. [Google Scholar]
  3. Liu, Y.; Zhang, W.; Yu, Y. Truth inference with a deep clustering-based aggregation model. IEEE Access 2020, 8, 16662–16675. [Google Scholar]
  4. Gil-Gonzalez, J.; Orozco-Gutierrez, A.; Alvarez-Meza, A. Learning from multiple inconsistent and dependent annotators to support classification tasks. Neurocomputing 2021, 423, 236–247. [Google Scholar] [CrossRef]
  5. Sung, H.E.; Chen, C.K.; Xiao, H.; Lin, S.D. A Classification Model for Diverse and Noisy Labelers. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining; Springer: Berlin/Heidelberg, Germany, 2017; pp. 58–69. [Google Scholar]
  6. Yan, Y.; Rosales, R.; Fung, G.; Subramanian, R.; Dy, J. Learning from multiple annotators with varying expertise. Mach. Learn. 2014, 95, 291–327. [Google Scholar] [CrossRef] [Green Version]
  7. Xu, G.; Ding, W.; Tang, J.; Yang, S.; Huang, G.Y.; Liu, Z. Learning effective embeddings from crowdsourced labels: An educational case study. In Proceedings of the 2019 IEEE 35th International Conference on Data Engineering (ICDE), Macao, China, 8–11 April 2019; pp. 1922–1927. [Google Scholar]
  8. Tanno, R.; Saeedi, A.; Sankaranarayanan, S.; Alexander, D.C.; Silberman, N. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 11244–11253. [Google Scholar]
  9. Davani, A.M.; Díaz, M.; Prabhakaran, V. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguist. 2022, 10, 92–110. [Google Scholar] [CrossRef]
  10. Kara, Y.E.; Genc, G.; Aran, O.; Akarun, L. Modeling annotator behaviors for crowd labeling. Neurocomputing 2015, 160, 141–156. [Google Scholar] [CrossRef]
  11. Cao, P.; Xu, Y.; Kong, Y.; Wang, Y. Max-mig: An information theoretic approach for joint learning from crowds. arXiv 2019, arXiv:1905.13436. [Google Scholar]
  12. Chen, Z.; Wang, H.; Sun, H.; Chen, P.; Han, T.; Liu, X.; Yang, J. Structured Probabilistic End-to-End Learning from Crowds. In Proceedings of the IJCAI, Yokohama, Japan, 7–21 January 2021; pp. 1512–1518. [Google Scholar]
  13. Ruiz, P.; Morales-Álvarez, P.; Molina, R.; Katsaggelos, A.K. Learning from crowds with variational Gaussian processes. Pattern Recognit. 2019, 88, 298–311. [Google Scholar] [CrossRef]
  14. G. Rodrigo, E.; Aledo, J.A.; Gámez, J.A. Machine learning from crowds: A systematic review of its applications. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2019, 9, e1288. [Google Scholar] [CrossRef]
  15. Zhang, P.; Obradovic, Z. Learning from inconsistent and unreliable annotators by a gaussian mixture model and bayesian information criterion. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Athens, Greece, 5–9 September 2011; pp. 553–568. [Google Scholar]
  16. Zhang, J. Knowledge learning with crowdsourcing: A brief review and systematic perspective. IEEE/CAA J. Autom. Sin. 2022, 9, 749–762. [Google Scholar] [CrossRef]
  17. Zhu, T.; Pimentel, M.A.; Clifford, G.D.; Clifton, D.A. Unsupervised Bayesian inference to fuse biosignal sensory estimates for personalizing care. IEEE J. Biomed. Health Inform. 2018, 23, 47–58. [Google Scholar] [CrossRef] [PubMed]
  18. Song, H.; Kim, M.; Park, D.; Shin, Y.; Lee, J.G. Learning from noisy labels with deep neural networks: A survey. IEEE Trans. Neural Netw. Learn. Syst. 2022. [Google Scholar] [CrossRef] [PubMed]
  19. Cheng, L.; Zhou, X.; Zhao, L.; Li, D.; Shang, H.; Zheng, Y.; Pan, P.; Xu, Y. Weakly supervised learning with side information for noisy labeled images. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020, Proceedings, Part XXX 16; Springer: Berlin/Heidelberg, Germany, 2020; pp. 306–321. [Google Scholar]
  20. Lee, K.; Yun, S.; Lee, K.; Lee, H.; Li, B.; Shin, J. Robust inference via generative classifiers for handling noisy labels. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 3763–3772. [Google Scholar]
  21. Chen, P.; Liao, B.B.; Chen, G.; Zhang, S. Understanding and utilizing deep neural networks trained with noisy labels. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 1062–1070. [Google Scholar]
  22. Yu, X.; Han, B.; Yao, J.; Niu, G.; Tsang, I.; Sugiyama, M. How does disagreement help generalization against label corruption? In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 7164–7173. [Google Scholar]
  23. Lyu, X.; Wang, J.; Zeng, T.; Li, X.; Chen, J.; Wang, X.; Xu, Z. TSS-Net: Two-stage with sample selection and semi-supervised net for deep learning with noisy labels. In Proceedings of the Third International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI 2022), SPIE, Guangzhou, China, 12–14 August 2022; Volume 12509, pp. 575–584. [Google Scholar]
  24. Shen, Y.; Sanghavi, S. Learning with bad training data via iterative trimmed loss minimization. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 5739–5748. [Google Scholar]
  25. Ghosh, A.; Manwani, N.; Sastry, P. Making risk minimization tolerant to label noise. Neurocomputing 2015, 160, 93–107. [Google Scholar] [CrossRef] [Green Version]
  26. Ghosh, A.; Kumar, H.; Sastry, P.S. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017; Volume 31. [Google Scholar]
  27. Albarqouni, S.; Baur, C.; Achilles, F.; Belagiannis, V.; Demirci, S.; Navab, N. Aggnet: Deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 2016, 35, 1313–1321. [Google Scholar] [CrossRef] [PubMed]
  28. Rodrigues, F.; Pereira, F. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; Volume 32. [Google Scholar]
  29. Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9, 2579–2605. [Google Scholar]
  30. Chattopadhay, A.; Sarkar, A.; Howlader, P.; Balasubramanian, V.N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 839–847. [Google Scholar]
  31. Rizos, G.; Schuller, B.W. Average jane, where art thou?–recent avenues in efficient machine learning under subjectivity uncertainty. In Proceedings of the International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, Lisbon, Portugal, 15–19 June 2020; pp. 42–55. [Google Scholar]
  32. Zhang, J.; Wu, X.; Sheng, V.S. Imbalanced multiple noisy labeling. IEEE Trans. Knowl. Data Eng. 2014, 27, 489–503. [Google Scholar] [CrossRef]
  33. Dawid, A.P.; Skene, A.M. Maximum likelihood estimation of observer error-rates using the EM algorithm. J. R. Stat. Soc. Ser. C (Appl. Stat.) 1979, 28, 20–28. [Google Scholar] [CrossRef]
  34. Raykar, V.C.; Yu, S.; Zhao, L.H.; Valadez, G.H.; Florin, C.; Bogoni, L.; Moy, L. Learning from crowds. J. Mach. Learn. Res. 2010, 11, 1297–1322. [Google Scholar]
  35. Groot, P.; Birlutiu, A.; Heskes, T. Learning from multiple annotators with Gaussian processes. In Proceedings of the International Conference on Artificial Neural Networks, Espoo, Finland, 14–17 June 2011; pp. 159–164. [Google Scholar]
  36. Xiao, H.; Xiao, H.; Eckert, C. Learning from multiple observers with unknown expertise. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Gold Coast, Australia, 14–17 April 2013; pp. 595–606. [Google Scholar]
  37. Morales-Alvarez, P.; Ruiz, P.; Coughlin, S.; Molina, R.; Katsaggelos, A.K. Scalable variational Gaussian processes for crowdsourcing: Glitch detection in LIGO. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 44, 1534–1551. [Google Scholar] [CrossRef]
  38. Gil-Gonzalez, J.; Alvarez-Meza, A.; Orozco-Gutierrez, A. Learning from multiple annotators using kernel alignment. Pattern Recognit. Lett. 2018, 116, 150–156. [Google Scholar] [CrossRef]
  39. Morales-Álvarez, P.; Ruiz, P.; Santos-Rodríguez, R.; Molina, R.; Katsaggelos, A.K. Scalable and efficient learning from crowds with Gaussian processes. Inf. Fusion 2019, 52, 110–127. [Google Scholar] [CrossRef]
  40. Rodrigues, F.; Pereira, F.; Ribeiro, B. Sequence labeling with multiple annotators. Mach. Learn. 2014, 95, 165–181. [Google Scholar] [CrossRef] [Green Version]
  41. Wang, X.; Bi, J. Bi-convex optimization to learn classifiers from multiple biomedical annotations. IEEE/ACM Trans. Comput. Biol. Bioinform. 2016, 14, 564–575. [Google Scholar] [CrossRef] [PubMed]
  42. Zhang, Z.; Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. Adv. Neural Inf. Process. Syst. 2018, 31, 1–11. [Google Scholar]
  43. Gil-González, J.; Valencia-Duque, A.; Álvarez-Meza, A.; Orozco-Gutiérrez, Á.; García-Moreno, A. Regularized chained deep neural network classifier for multiple annotators. Appl. Sci. 2021, 11, 5409. [Google Scholar] [CrossRef]
  44. Zhao, X.; Li, X.; Bi, D.; Wang, H.; Xie, Y.; Alhudhaif, A.; Alenezi, F. L1-norm constraint kernel adaptive filtering framework for precise and robust indoor localization under the internet of things. Inf. Sci. 2022, 587, 206–225. [Google Scholar] [CrossRef]
  45. Box, G.E.; Cox, D.R. An analysis of transformations. J. R. Stat. Soc. Ser. B (Methodol.) 1964, 26, 211–243. [Google Scholar] [CrossRef]
  46. Saul, A.; Hensman, J.; Vehtari, A.; Lawrence, N. Chained Gaussian processes. In Proceedings of the Artificial Intelligence and Statistics, Cadiz, Spain, 9–11 May 2016; pp. 1431–1440. [Google Scholar]
  47. Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems; O’Reilly Media: Sebastopol, CA, USA, 2019. [Google Scholar]
  48. Xiao, H.; Rasul, K.; Vollgraf, R. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv 2017, arXiv:1708.07747. [Google Scholar]
  49. Hernández-Muriel, J.A.; Bermeo-Ulloa, J.B.; Holguin-Londoño, M.; Álvarez-Meza, A.M.; Orozco-Gutiérrez, Á.A. Bearing health monitoring using relief-F-based feature relevance analysis and HMM. Appl. Sci. 2020, 10, 5170. [Google Scholar] [CrossRef]
  50. LeCun, Y.; Boser, B.; Denker, J.; Henderson, D.; Howard, R.; Hubbard, W.; Jackel, L. Handwritten digit recognition with a back-propagation network. Adv. Neural Inf. Process. Syst. 1989, 2, 396–404. [Google Scholar]
  51. Dogs vs. Cats—Kaggle.com. Available online: https://www.kaggle.com/c/dogs-vs-cats (accessed on 6 January 2023).
  52. Peterson, J.C.; Battleday, R.M.; Griffiths, T.L.; Russakovsky, O. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27–28 October 2019; pp. 9617–9626. [Google Scholar]
  53. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
  54. Tzanetakis, G.; Cook, P. Musical genre classification of audio signals. IEEE Trans. Speech Audio Process. 2002, 10, 293–302. [Google Scholar] [CrossRef]
  55. Rodrigues, F.; Pereira, F.; Ribeiro, B. Learning from multiple annotators: Distinguishing good from random labelers. Pattern Recognit. Lett. 2013, 34, 1428–1436. [Google Scholar] [CrossRef]
  56. Gil-Gonzalez, J.; Giraldo, J.J.; Alvarez-Meza, A.; Orozco-Gutierrez, A.; Alvarez, M. Correlated Chained Gaussian Processes for Datasets with Multiple Annotators. IEEE Trans. Neural Netw. Learn. Syst. 2021. [Google Scholar] [CrossRef]
  57. MacKay, D.J.; Mac Kay, D.J. Information Theory, Inference and Learning Algorithms; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
  58. Demšar, J. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 2006, 7, 1–30. [Google Scholar]
  59. Li, Z.L.; Zhang, G.W.; Yu, J.; Xu, L.Y. Dynamic Graph Structure Learning for Multivariate Time Series Forecasting. Pattern Recognit. 2023, 138, 109423. [Google Scholar] [CrossRef]
  60. Leroux, L.; Castets, M.; Baron, C.; Escorihuela, M.J.; Bégué, A.; Seen, D.L. Maize yield estimation in West Africa from crop process-induced combinations of multi-domain remote sensing indices. Eur. J. Agron. 2019, 108, 11–26. [Google Scholar] [CrossRef]
  61. Montavon, G.; Binder, A.; Lapuschkin, S.; Samek, W.; Müller, K.R. Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning; Springer Nature: Cham, Switzerland, 2019; pp. 193–209. [Google Scholar]
  62. Holzinger, A.; Saranti, A.; Molnar, C.; Biecek, P.; Samek, W. Explainable AI methods-a brief overview. In Proceedings of the xxAI-Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, Vienna, Austria, 18 July 2020, Revised and Extended Papers; Springer: Berlin/Heidelberg, Germany, 2022; pp. 13–38. [Google Scholar]
  63. Bennetot, A.; Donadello, I.; Qadi, A.E.; Dragoni, M.; Frossard, T.; Wagner, B.; Saranti, A.; Tulli, S.; Trocan, M.; Chatila, R.; et al. A practical tutorial on explainable ai techniques. arXiv 2021, arXiv:2111.14260. [Google Scholar]
  64. Saranti, A.; Hudec, M.; Mináriková, E.; Takáč, Z.; Großschedl, U.; Koch, C.; Pfeifer, B.; Angerschmid, A.; Holzinger, A. Actionable Explainable AI (AxAI): A Practical Example with Aggregation Functions for Adaptive Classification and Textual Explanations for Interpretable Machine Learning. Mach. Learn. Knowl. Extr. 2022, 4, 924–953. [Google Scholar] [CrossRef]
Sensors 23 03518 g001 550
Figure 1. GCECDL sketch for multiple-annotator classification holding ground truth prediction and instance-based expert reliability estimation.
Figure 1. GCECDL sketch for multiple-annotator classification holding ground truth prediction and instance-based expert reliability estimation.
Sensors 23 03518 g001
Sensors 23 03518 g002 550
Figure 2. GCECDL-based fully connected architecture for tabular databases. F C stands for a fully connected (dense) layer. F C 1 and F C 2 use a selu activation. A dropout layer (Dropout) is included to avoid overfitting. The MA layer contains two fully connected layers that output the hidden ground truth label and the annotator’s reliability, fixing a softmax and a sigmoid activation, respectively.
Figure 2. GCECDL-based fully connected architecture for tabular databases. F C stands for a fully connected (dense) layer. F C 1 and F C 2 use a selu activation. A dropout layer (Dropout) is included to avoid overfitting. The MA layer contains two fully connected layers that output the hidden ground truth label and the annotator’s reliability, fixing a softmax and a sigmoid activation, respectively.
Sensors 23 03518 g002
Sensors 23 03518 g003 550
Figure 3. GCECDL-based convolutional architecture for image databases. Four convolutional layers with 3 × 3 patches, 2 × 2 max pooling, and ReLU activations are included. The MA layer outputs the hidden ground truth label and the annotator’s reliability fixing a softmax and a sigmoid activation, respectively.
Figure 3. GCECDL-based convolutional architecture for image databases. Four convolutional layers with 3 × 3 patches, 2 × 2 max pooling, and ReLU activations are included. The MA layer outputs the hidden ground truth label and the annotator’s reliability fixing a softmax and a sigmoid activation, respectively.
Sensors 23 03518 g003
Sensors 23 03518 g004 550
Figure 4. Synthetic results—1D dataset. The predictive label’s probability (PLP) is shown, comparing the prediction of our GCECDL (AUC = 0.99) against: DL-CL(MW) (AUC = 0.79), DL-MV (AUC = 0.9), and RCDNN (AUC = 0.99). Label classes are represented as blue, green, and orange.
Figure 4. Synthetic results—1D dataset. The predictive label’s probability (PLP) is shown, comparing the prediction of our GCECDL (AUC = 0.99) against: DL-CL(MW) (AUC = 0.79), DL-MV (AUC = 0.9), and RCDNN (AUC = 0.99). Label classes are represented as blue, green, and orange.
Sensors 23 03518 g004
Sensors 23 03518 g005 550
Figure 5. GCECDL-based annotators’ performance (reliability) estimation for the synthetic 1D experiment. Orange dashed line depicts (from left to right) the simulated accuracy for each annotator based on Equations (19)–(22). The blue line shows (from left to right) the estimated annotator’s reliability ( λ r ).
Figure 5. GCECDL-based annotators’ performance (reliability) estimation for the synthetic 1D experiment. Orange dashed line depicts (from left to right) the simulated accuracy for each annotator based on Equations (19)–(22). The blue line shows (from left to right) the estimated annotator’s reliability ( λ r ).
Sensors 23 03518 g005
Sensors 23 03518 g006 550
Figure 6. Simulated annotators’ for the MNIST database. A 2D t-SNE-based projection of the Mnist training set from the FC1 layer feature maps is shown (some exemplary images are depicted over their 2D projections, where numbers from zero to nine stand for each MNIST class). The ground truth labels are used as a color. The second row depicted the 2D projection of each annotator’s labels in color. The third row presents the confusion matrix of each expert and their achieved accuracy.
Figure 6. Simulated annotators’ for the MNIST database. A 2D t-SNE-based projection of the Mnist training set from the FC1 layer feature maps is shown (some exemplary images are depicted over their 2D projections, where numbers from zero to nine stand for each MNIST class). The ground truth labels are used as a color. The second row depicted the 2D projection of each annotator’s labels in color. The third row presents the confusion matrix of each expert and their achieved accuracy.
Sensors 23 03518 g006
Sensors 23 03518 g007 550
Figure 7. Visual inspection results for the MNIST database. A 2D t-SNE-based projection of the Mnist test set from the FC1 layer feature maps is shown (some exemplary images are depicted over their 2D projections, where numbers from zero to nine stand for each MNIST class). The ground truth labels are shown in color. The second row depicted the 2D projection of the ground truth prediction in color with some explanation maps for visualization purposes. The third row presents the 2D projection of each annotator’s reliability estimation in color.
Figure 7. Visual inspection results for the MNIST database. A 2D t-SNE-based projection of the Mnist test set from the FC1 layer feature maps is shown (some exemplary images are depicted over their 2D projections, where numbers from zero to nine stand for each MNIST class). The ground truth labels are shown in color. The second row depicted the 2D projection of the ground truth prediction in color with some explanation maps for visualization purposes. The third row presents the 2D projection of each annotator’s reliability estimation in color.
Sensors 23 03518 g007
Table
Table 1. Description of the number of features (P), instances (N), and classes (K) of tested datasets for multiple-annotator classification.
Table 1. Description of the number of features (P), instances (N), and classes (K) of tested datasets for multiple-annotator classification.
NameInput Shape (P)Number of Instances (N)Number of Classes (K)
1D Synthetic15003
Ocupancy720,5602
Skin4245,0572
Tic-Tac-Toe99582
Balance46253
Iris41503
New Thyroid52153
Wine131783
Fashion-Mnist78470,00010
Segmentation1823107
Western734134
Cat vs. Dog200 × 200 × 325,0002
Mnist28 × 28 × 170,00010
CIFAR-10H32 × 32 × 319,23310
LabelMe256 × 256 × 326888
Music124100010
Table
Table 2. A brief overview of the state-of-the-art methods tested.
Table 2. A brief overview of the state-of-the-art methods tested.
AlgorithmDescription
DL-GOLDA DL classification model using the real labels (upper bound).
DL-MVA DL classification model using the MV of the labels as the ground truth.
RCDNN [43]A regularized chained deep neural network which predicts the ground truth and annotators’ performance from input space samples.
DL-CL(MW) [28]A crowd Layer for DL, where annotators’ parameters are constant across the input space.
Table
Table 3. Method comparison results—multiple-annotator classification. Bold: the highest score for each measure excluding the upper bound (target) classifier DL-GOLD. For our GCECDL, the fixed q-value is presented in parentheses. Friedman test: we obtain a Chi-square of 27.08, 23.48, 28.03, and 7.92 (p-value = 5.6 × 10 − 6 , 3.56 × 10 − 6 , 3.20 × 10 − 5 , and 0.047) for ACC, BACC, NMI, and AUC measures, respectively.
Table 3. Method comparison results—multiple-annotator classification. Bold: the highest score for each measure excluding the upper bound (target) classifier DL-GOLD. For our GCECDL, the fixed q-value is presented in parentheses. Friedman test: we obtain a Chi-square of 27.08, 23.48, 28.03, and 7.92 (p-value = 5.6 × 10 − 6 , 3.56 × 10 − 6 , 3.20 × 10 − 5 , and 0.047) for ACC, BACC, NMI, and AUC measures, respectively.
DatabaseMeasureDL-GOLDDL-CL(MW)DL-MVRCDNNGCECDL
OcupancyACC [%]97.59 ± 0.1190.89 ± 0.9290.31 ± 0.8096.61 ± 0.4297.50 ± 0.06 (0.1)
BACC [%]95.92 ± 0.0974.30 ± 3.0475.50 ± 0.9993.22 ± 1.2795.86 ± 0.10 (0.1)
NMI [%]83.91 ± 0.3557.98 ± 2.1652.32 ± 3.5677.93 ± 2.5183.64 ± 0.31 (0.1)
AUC [%]97.96 ± 0.0587.17 ± 1.5287.75 ± 0.4999.03 ± 0.1497.92 ± 0.05 (0.1)
SkinACC [%]99.74 ± 0.1356.48 ± 0.3680.55 ± 3.2389.45 ± 2.0696.06 ± 0.59 (0.1)
BACC [%]99.62 ± 0.1545.07 ± 0.4679.80 ± 1.7964.27 ± 10.6690.60 ± 4.31 (0.1)
NMI [%]96.86 ± 1.1518.31 ± 0.2113.56 ± 1.0638.99 ± 8.2571.14 ± 5.22 (0.1)
AUC [%]99.81 ± 0.0891.80 ± 1.0380.45 ± 6.2282.13 ± 5.3399.30 ± 0.10 (0.1)
Tic-Tac-ToeACC [%]98.47 ± 0.1780.00 ± 3.7378.26 ± 1.8590.49 ± 17.1697.28 ± 1.17 (0.2)
BACC [%]96.00 ± 0.2766.24 ± 5.9856.09 ± 2.1478.74 ± 38.9393.10 ± 2.78 (0.2)
NMI [%]88.57 ± 1.5236.88 ± 6.0723.90 ± 2.0571.42 ± 24.7983.52 ± 6.13 (0.2)
AUC [%]98.05 ± 0.1496.48 ± 1.0085.37 ± 1.2191.71 ± 21.9396.55 ± 1.39 (0.2)
BalanceACC [%]91.91 ± 2.5085.27 ± 1.9373.72 ± 1.5191.33 ± 1.1490.21 ± 0.83 (0.1)
BACC [%]60.00 ± 5.6874.9 ± 4.0943.39 ± 0.846.93 ± 1.0075.27 ± 7.33 (0.1)
NMI [%]73.34 ± 3.3261.98 ± 3.5859.43 ± 2.0466.52 ± 3.3068.00 ± 2.19 (0.1)
AUC [%]83.66 ± 4.9184.97 ± 4.0870.66 ± 3.3564.51 ± 17.3681.11 ± 8.25 (0.1)
IrisACC [%]97.34 ± 0.9495.11 ± 2.5995.33 ± 1.5996.89 ± 1.1597.34 ± 0.94 (0.1)
BACC [%]96.47 ± 1.2491.67 ± 4.5491.56 ± 1.8494.24 ± 2.0396.06 ± 2.11 (0.1)
NMI [%]91.69 ± 1.9186.82 ± 5.5987.61 ± 2.8689.68 ± 3.7491.15 ± 3.07 (0.1)
AUC [%]100.00 ± 0.00100.00 ± 0.00100.00 ± 0.0099.83 ± 0.00100.00 ± 0.00 (0.1)
New ThyroidACC [%]96.15 ± 0.7795.69 ± 1.1597.69 ±1.0395.08 ± 0.6296.00 ± 1.84 (0.01)
BACC [%]92.5 ± 1.9492.61 ± 2.893.88 ±2.6588.41 ± 2.2290.52 ± 4.36 (0.01)
NMI [%]82.98 ± 3.4482.38 ± 3.8690.44 ±3.6180.40 ± 2.4283.61 ± 6.85 (0.01)
AUC [%]96.19 ± 0.9695.46 ± 1.7897.80 ± 0.694.72 ± 1.6795.19 ± 2.23 (0.01)
WineACC [%]97.59 ± 0.8595.74 ± 1.6773.15 ± 3.7394.26 ± 2.1096.48 ± 2.38 (0.01)
BACC [%]96.59 ± 1.0894.17 ± 2.5264.44 ± 5.4291.99 ± 2.5996.68 ± 2.92 (0.01)
NMI [%]91.50 ± 3.0785.54 ± 4.0344.18 ± 4.3982.91 ± 5.9789.87 ± 5.37 (0.01)
AUC [%]98.88 ± 0.3799.88 ± 0.3787.76 ± 2.8897.13 ± 1.26100.0 ± 0.00 (0.01)
F-MnistAcc [%]86.45 ± 0.2579.54 ± 9.1685.83 ± 0.3586.98 ± 0.7188.26 ± 0.26 (0.01)
BACC [%]84.92 ± 0.2777.28 ± 10.1184.23 ± 0.3885.50 ± 0.7786.94 ± 0.28 (0.01)
NMI [%]77.16 ± 0.2172.92 ± 3.8476.46 ± 0.3878.75 ± 0.4879.89 ± 0.21 (0.01)
AUC [%]88.75 ± 1.6185.61 ± 13.9590.42 ± 1.4689.33 ± 2.9389.11 ± 2.27 (0.01)
SegmentationACC [%]95.17 ± 0.2594.65 ± 0.9591.11 ± 0.9491.85 ± 0.6794.88 ± 0.8 (0.01)
BACC [%]94.49 ± 0.5993.81 ± 1.0989.79 ± 1.0990.69 ± 0.7794.15 ± 0.95 (0.01)
NMI [%]91.07 ± 0.7790.38 ±1.1884.46 ± 1.6586.27 ± 0.7089.92 ± 1.25 (0.01)
AUC [%]99.95 ± 0.1399.76 ± 0.2098.71 ± 0.4498.94 ± 0.3399.79 ± 0.22 (0.01)
WesternACC [%]99.25 ± 0.3895.94 ± 4.3995.11 ± 1.1597.89 ± 0.4098.31 ± 0.61 (0.01)
BACC [%]99.22 ± 0.4294.37 ± 5.5493.47 ± 1.4597.25 ± 0.5798.00 ± 0.97 (0.01)
NMI [%]98.61 ± 0.7096.80 ± 1.7094.33 ± 0.9797.16 ± 0.3097.35 ± 0.60 (0.01)
AUC [%]100.00 ± 0.00100.00 ± 0.0098.97 ± 0.8599.99 ± 0.01100.00 ± 0.00 (0.01)
Cats vs. DogsACC [%]85.00 ± 0.4468.97 ± 0.8458.69 ± 1.0270.84 ± 4.7174.15 ± 0.58 (0.1)
BACC [%]70.19 ± 0.2237.44 ± 0.3516.50 ± 0.5143.51 ± 2.0848.24 ± 0.54 (0.1)
NMI [%]40.66 ± 0.4111.03 ± 0.252.50 ± 0.1216.02 ± 1.0420.79 ± 0.33 (0.1)
AUC [%]93.17 ± 0.2268.94 ± 0.8058.63 ± 0.9870.98 ± 4.5782.99 ± 0.30 (0.1)
MnistACC [%]99.32 ± 0.0687.99 ± 2.7392.88 ± 0.5499.09 ± 0.0599.11 ± 0.08 (0.01)
BACC [%]99.22 ± 0.0686.49 ± 3.1191.97 ± 0.5998.98 ± 0.0699.02 ± 0.09 (0.01)
NMI [%]97.95 ± 0.1582.82 ± 2.2983.43 ± 1.0297.28 ± 0.1197.39 ± 0.21 (0.01)
AUC [%]99.81 ± 0.0899.82 ± 0.0297.88 ± 0.3399.71 ± 0.0699.68 ± 0.08 (0.01)
CIFAR-10HACC [%]71.72 ± 1.1260.80 ± 1.5968.24 ± 1.0569.53 ± 0.6369.24 ± 0.67 (0.01)
BACC [%]68.46 ± 0.2856.56 ± 1.7264.69 ± 1.1866.18 ± 0.7265.88 ± 0.76 (0.01)
NMI [%]64.2 ± 0.2343.09 ± 1.2149.55 ± 1.451.44 ± 0.8150.95 ± 0.78 (0.01)
AUC [%]96.08 ± 0.1189.42 ± 0.794.81 ± 0.3595.17 ± 0.1695.01 ± 0.19 (0.01)
LabelMeACC [%]90.91 ± 0.4483.11 ± 0.9676.94 ± 1.1589.09 ± 0.4188.97 ± 0.55 (0.01)
BACC [%]90.03 ± 0.4281.85 ± 0.8675.08 ± 1.2588.01 ± 0.4287.97 ± 0.57 (0.01)
NMI [%]81.36 ± 0.7973.92 ± 0.8068.36 ± 0.5477.82 ± 0.5578.00 ± 0.70 (0.01)
AUC [%]99.37 ± 0.0696.34 ± 0.8897.65 ± 0.1399.14 ± 0.0699.11 ± 0.08 (0.01)
MusicACC [%]76.4 ± 0.8857.13 ± 2.8062.8 ± 1.0665.80 ± 2.8365.63 ± 2.41 (0.01)
BACC [%]74.32 ± 0.9151.47 ± 2.9157.67 ± 1.2761.96 ± 3.3261.83 ± 2.84 (0.01)
NMI [%]65.86 ± 1.1657.25 ± 1.5553.87 ± 1.4957.02 ± 2.6357.36 ± 1.40 (0.01)
AUC [%]96.34 ± 0.1388.62 ± 3.0893.58 ± 0.2194.07 ± 0.9494.13 ± 0.50 (0.01)
Average RankingACC [%] − − ± − − 3.26 ± − − 3.40 ± − − 2.00 ± − − 1.33 ± − −
BACC [%] − − ± − − 3.00 ± − − 3.40 ± − − 2.33 ± − − 1.26 ± − −
NMI [%] − − ± − − 3.14 ± − − 3.46 ± − − 2.21 ± − − 1.20 ± − −
AUC [%] − − ± − − 2.67 ± − − 3.13 ± − − 2.47 ± − − 1.73 ± − −
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Triana-Martinez, J.C.; Gil-González, J.; Fernandez-Gallego, J.A.; Álvarez-Meza, A.M.; Castellanos-Dominguez, C.G. Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification. Sensors 2023, 23, 3518. https://doi.org/10.3390/s23073518

AMA Style

Triana-Martinez JC, Gil-González J, Fernandez-Gallego JA, Álvarez-Meza AM, Castellanos-Dominguez CG. Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification. Sensors. 2023; 23(7):3518. https://doi.org/10.3390/s23073518

Chicago/Turabian Style

Triana-Martinez, Jenniffer Carolina, Julian Gil-González, Jose A. Fernandez-Gallego, Andrés Marino Álvarez-Meza, and Cesar German Castellanos-Dominguez. 2023. "Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification" Sensors 23, no. 7: 3518. https://doi.org/10.3390/s23073518

APA Style

Triana-Martinez, J. C., Gil-González, J., Fernandez-Gallego, J. A., Álvarez-Meza, A. M., & Castellanos-Dominguez, C. G. (2023). Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification. Sensors, 23(7), 3518. https://doi.org/10.3390/s23073518

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop