Leveraging Task Variability in Meta-learning

Meta-learning (ML) utilizes extracted meta-knowledge from data to enable models to perform well on unseen data that they have not encountered before. Typically, this meta-knowledge is acquired from randomly sampled task batches, and a critical assumption in the meta-learning is that all tasks in a batch equally contribute to the meta-knowledge. However, this assumption may not always hold true. In this study, we explore the impact of weighting tasks in a batch based on their contribution to meta-knowledge. We achieve this by introducing a learnable “task attention module” that can be integrated into any episodic training pipeline. We demonstrate that our approach improves the quality of the meta-knowledge obtained on standard meta-learning benchmarks such as miniImagenet, FC100, and tieredImagenet, as well as on noisy and cross-domain few-shot benchmarks. Additionally, we conduct a comprehensive analysis of the proposed task attention module to gain insights into its operation.

Not applicable.

1. https://github.com/taskattention/task-attended-metalearning.git.

Authors and Affiliations

1. Indian Institute of Technology, Ropar, India

   Aroof Aimen & Bharat Ladrecha

2. University of Texas, Dallas, USA

   Sahil Sidheekh

3. Indian Institute of Technology, Palakkad, India

   Narayanan C. Krishnan

Corresponding authors

Correspondence to Aroof Aimen or Narayanan C. Krishnan.

Conflict of interest

With the submission of this manuscript, we would like to declare that: We do not have any commercial or non-commercial conflicts of interest associated with this work. This manuscript has not been published elsewhere and is not under consideration by another journal. All authors have approved the manuscript and agreed upon the submission to SN Journal (Special Issue: Research Trends in Computational Intelligence). This work has no ethical implications, and all the results are reproducible. The link to the code is provided in the manuscript. The support and the resources provided by ‘PARAM Shivay Facility’ under the National Supercomputing Mission, Government of India at the Indian Institute of Technology, Varanasi, and under the Google Tensorflow Research award are gratefully acknowledged.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Research Trends in Computational Intelligence” guest edited by Anshul Verma, Pradeepika Verma, Vivek Kumar Singh and S. Karthikeyan.

Appendix A. Preliminary

A.0.1. Meta-knowledge as an Optimal Initialization

When meta-knowledge is a generic initialization on the model parameters learned through the experience over various tasks, it is enforced to be close to each individual training tasks’ optimal parameters. A model initialized with such an optimal prior quickly adapts to unseen tasks from the same distribution during meta-testing. MAML [12] employs a nested iterative process to learn the task-agnostic optimal prior \(\theta \). In the inner iterations representing the task adaptation steps, \(\theta \) is separately fine-tuned for each meta-training task \(\mathcal {T}_i\) of a batch using \(D_i\) to obtain \(\phi _i\) through gradient descent on the train loss L using learning rate \(\alpha \). Specifically, \(\phi _i\) is initialized as \(\theta \) and updated using \(\phi _i \leftarrow \phi _i - \alpha \nabla _{\phi _i}L(\phi _i)\), T times resulting in the adapted model \(\phi _i^T\). In the outer loop, meta-knowledge is gathered by optimizing \(\theta \) over loss \(L^*\) computed with the task adapted model parameters \(\phi _i^T\) on query dataset \(D^*_i\). Specifically, during meta-optimization \(\theta \leftarrow \theta -\beta \nabla _{\theta } \sum _{i=1}^B L^*(\phi _i^T)\) using a task batch of size B and learning rate \(\beta \). MetaSGD [24] improves upon MAML by learning parameter-specific learning rates \(\varvec{\alpha }\) in addition to the optimal initialization in a similar nested iterative procedure. Meta-knowledge is gathered by optimizing \(\theta \) and \(\varvec{\alpha }\) in the outer loop using the loss \(L^*\) computed on query set \(D^*_i\). Specifically, during meta-optimization \((\theta ,\varvec{\alpha }) \leftarrow (\theta ,\varvec{\alpha }) - \beta \nabla _{(\theta , \varvec{\alpha })} \sum _{i=1}^B L^*(\phi _i^T)\). Learning dynamic learning rates for each parameter of a model makes MetaSGD faster and more generalizable than MAML. A single adaptation step is sufficient to adjust the model towards a new task. The performance of MAML is attributed to the reuse of the features across tasks rather than the rapid learning of new tasks [31]. Exploiting this characteristic, ANIL freezes the feature backbone layers (\(1,\dots , l-1\)) and only adapts classifier layer (l) in the inner loop T times. Specifically during adaptation \(\phi _i^l \leftarrow \phi _i^l - \alpha \nabla _{\phi _i^l} L(\phi _i^l)\). During meta-optimization \(\theta ^{1,\dots , l} \leftarrow \theta ^{1,\dots , l} -\beta \nabla _{\theta ^{1,\dots , l}} \sum _{i=1}^B L^*(\phi _i^{lT})\) i.e., all layers are learned in the outer loop. Freezing the feature backbone during adaptation reduces the overhead of computing gradient through the gradient (differentiating through the inner loop), and thereby heavier backbones could be used for the feature extraction. TAML [16] suggests that the optimal prior learned by MAML may still be biased towards some tasks. They propose to reduce this bias and enforce equity among the tasks by explicitly minimizing the inequality among the performances of tasks in a batch. The inequality defined using statistical measures such as Theil Index, Atkinson Index, Generalized Entropy Index, and Gini Coefficient among the performances of tasks in a batch is used as a regularizer while gathering the meta-knowledge. For the baseline comparison, in our experiments, we use the Theil index for TAML owing to its average best results. Specifically during meta-optimization \(\theta \leftarrow \theta -\beta \nabla _{\theta } \left[ \sum _{i=1}^B L^*(\phi _i^T)+ \lambda \left\{ \dfrac{ L^*(\phi _i^0)}{\bar{L}^*(\phi _i^0)} \ln \dfrac{L^*(\phi _i^0)}{\bar{L}^*(\phi _i^0)}\right\} \right] \) (for TAML-Theil Index) where B is the number of tasks in a batch, \(L^*(\phi _i^0)\) is the loss incurred by initial model \(\phi _i^0\) on the query set \({D}^*_i\) of task \(\mathcal {T}_i\) and \(\bar{L}^{*}(\phi _i^0)\) is the average query loss of initial model on a batch of tasks. As TAML enforces equity of the optimal prior towards meta-train tasks, it counters the adaptation, which leads to slow and unstable training largely dependent on \(\lambda \).

A.0.2. Meta-knowledge as a Parametric Optimizer

A regulated gradient-based optimizer gathers the task-specific and task-agnostic meta-knowledge to traverse the loss surfaces of tasks in the meta-train set during meta-training. A base model guided by such a learned parametric optimizer quickly finds the way to minima even for unseen tasks sampled from the same distribution during meta-testing. MetaLSTM [32] is a recurrent parametric optimizer \(\theta \) that mimics the gradient-based optimization of a base model \(\phi \). This recurrent optimizer is an LSTM [15] and is inherently capable of performing two-level learning due to its architecture. During adaptation of \(\phi _i\) on \(D_i\), \(\theta \) takes meta information of \(\phi _i\) characterized by its current loss L and gradients \(\nabla _{\phi _ {i}}(L)\) as input and outputs the next set of parameters for \(\phi _i\). This adaptation procedure is repeated T times resulting in the adapted base-model \(\phi _i^T\). Internally, the cell state of \(\theta \) corresponds to \(\phi _i\), and the cell state update for \(\theta \) resembles a learned and controlled gradient update. The emphasis on previous parameters and the current update is regulated by the learned forget and input gates respectively. While adapting \(\phi _i\) to \(D_i\), information about the trajectory on the loss surface across the adaptation steps is captured in the hidden states of \(\theta \), representing the task-specific knowledge. During meta-optimization, \(\theta \) is updated based on the loss of the adapted model \(L^*(\phi _i^T)\) computed on the query set \(D^*_i\) to garner the meta-knowledge across tasks. Specifically, during meta-optimization, \(\theta \leftarrow \theta -\beta \nabla _{\theta } L^*(\phi _i^T)\). MetaLSTM updates parametric optimizer \(\theta \) after adapting the base model \(\phi \) to each task. This causes \(\theta \) to follow optima’s of all adapted base models leading to its elongated and fluctuating optimization trajectory, which is biased towards the last task. MetaLSTM++ [2] circumvents these issues as \(\theta \) is updated by an aggregate query loss of the adapted models on a batch of tasks. Batch updates smoothen the optimization trajectory of \(\theta \) and eliminate its bias towards the last task. Specifically, during meta-optimization \(\theta \leftarrow \theta -\beta \nabla _{\theta } \sum _{i=1}^B L^*(\phi _i^T)\).

Appendix B. Details of Proposed Approach

[Best viewed in color] Workflow of proposed training curriculum

Full size image

We explain the proposed approach through Figs. 1, 7, Algorithm 1, and equations. We first sample a batch of tasks (B) from a random pool of data (Fig. 7-Label \(\textcircled {1}\)). For each task, the base-model \(\phi _i\) is adapted using the support data \(D_i\) for T time-steps (line 7 and lines 20–32 in Algorithm 1, Fig. 7-Label \(\textcircled {3}\)). Specifically, the adaptation is done using gradient descent on the train loss L for initialization approaches (lines 22–26 in Algorithm 1, Fig. 7-GD), or the current loss and gradients are inputted to the meta-model \(\theta \) for optimization approaches, which then outputs the updated base-model parameters (lines 27–31 in Algorithm 1, Fig. 7-PO). The meta-information (\(\mathcal {I}\)) corresponding to each task in the batch is then calculated (Fig. 7-Label \(\textcircled {4}\)), which includes the loss, accuracy, loss-ratio, and gradient norm of adapted models on the query data. This is given as input to the task attention module (Fig. 1-Label \(\textcircled {2}\), Fig. 7-Label \(\textcircled {5}\)), which outputs the attention vector (line 10 in Algorithm 1, Fig. 7-Label \(\textcircled {6}\)). The attention vector and test losses are used to update the meta-model parameters \(\theta \) according to Eq. (2) (line 11 in Algorithm 1, Fig. 1-Label \(\textcircled {4}\), Fig. 7-Label \(\textcircled {7}\)). A new batch of tasks is then sampled and the base-models are adapted using the updated meta-model (Lines 12–16 in Algorithm 1, Fig. 1-Label \(\textcircled {5}\)). The mean test loss over the adapted base-models is calculated and used to update the parameters of the task attention module \(\delta \) according to Eq. (3).

Appendix C. Experiments

C.0.1. Datasets Details

miniImagenet dataset [41] comprises 600 color images of size 84 \(\times \) 84 from each of 100 classes sampled from the Imagenet dataset. The 100 classes are split into 64, 16 and 20 classes for meta-training, meta-validation and meta-testing respectively. miniImagenet-noisy [42] is constructed from the miniImagenet dataset with the additional constraint that tasks have noisy support labels and clean query labels. The noise in support labels is introduced by symmetry flipping, and the default noise ratio is 0.6. Fewshot Cifar 100 (FC100) dataset [30] has been created from Cifar 100 object classification dataset. It contains 600 color images of size 32 \(\times \) 32 corresponding to each of 100 classes grouped into 20 super-classes. Among 100 classes, 60 classes belonging to 12 super-classes correspond to the meta-train set, 20 classes from 4 super-classes to the meta-validation set, and the rest to the meta-test set. tieredImagenet [33] is a more challenging benchmark for few-shot image classification. It contains 779,165 color images sampled from 608 classes of Imagenet and are grouped into 34 super-classes. These super-classes are divided into 20, 6, and 8 disjoint sets for meta-training, meta-validation, and meta-testing. Metadataset [40] comprises of 10 freely available diverse datasets—Aircraft, CUB-200-2011, Describable Textures, Fungi, ILSVRC-2012, MSCOCO, Omniglot, Quick Draw, Traffic Signs, and VGG Flower. We utilized CUB-200, FGVC-Aircraft, Describable Textures, and Omniglot datasets from Metadataset. VTAB dataset [43] is a more diverse dataset than Metadataset that was proposed to avoid overlapping classes of sub-datasets with the Imagenet dataset. VTAB comprises of 19 datasets divided into three domains—Natural, Specialized, and Structured, depending on the type of images. The natural group contains Caltech101, CIFAR100, DTD, Flowers102, Pets, Sun397, and SVHN sub-datasets, while the specialized group consists of remote sensing datasets like EuroSAT and Resisic 45 and medical datasets like Retinopathy and Patch Camelyon. Structured contains object counting or 3D depth prediction datasets like Clevr/count, Clevr/distance, dSprites/location, dSprites/orientation, SmallNORB/azimuth, SmallNORB/elevation, DMLab, and KITTI/distance. We considered Natural sub-datasets like DTD, CIFAR FC 100, Flowers102, and SVHN, specialized sub-datasets like EuroSAT and Resisc45, and structured sub-datasets like dSprites_location and dSprites_orientation for cross-domain experimentation. According to [11], we have kept Describable Textures as a part of Metadataset and Flowers102 as a component of the VTAB dataset.

Rank analysis of tasks for maximum and minimum values of: loss, loss-ratio, accuracy and gradient norm throughout the training of TA-MAML\(^*\) for 5 way 1 shot setting on miniImagenet dataset

Full size image

C.0.2. Ablation Studies

We analyze the ranks of the tasks for maximum and minimum values of: loss, loss ratio, accuracy, and gradient norm in a batch wrt attention weights throughout meta-training of TA-MAML on a 5 way 1 and 5 shot settings on miniImagenet dataset (Figs. 8, 9). Specifically, the highest weighted task is given rank one, and the least weighted task in a batch is given the last rank. We observe that the TA module does not assign maximum weight to the tasks with maximum or minimum values of: test loss, loss ratio, gradient norm or accuracy throughout meta-training. Thus, the TA module does not trivially learn to assign weights to the tasks based on some component of meta-information but learns useful latent information from all the components to assign importance for the tasks in a batch.

Rank analysis of tasks for maximum and minimum values of: loss, loss-ratio, accuracy and gradient norm throughout the training of TA-MAML\(^*\) for 5 way 5 shot setting on miniImagenet dataset

Full size image

Trend analysis of weighted loss across meta-training iterations for TA-MAML\(^*\) on 5 way 1 shot (left) and 5 shot (right) settings on miniImagenet dataset. Iterations are in thousands

Full size image

C.0.3. Relation of Weights with Meta-information

In Fig. 10, we illustrate the trend of mean weighted loss across iterations for TA-MAML on 5 way 1 and 5 shot settings on miniImagenet dataset. The trend indicates that the average weighted loss decreases over the meta-training iterations. The shaded region represents a 95% confidence interval over 100 tasks.

C.0.4. Hyperparameter Details

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions