This repository contains the source code for the EMNLP 2022 (Findings) paper: Task Compass: Scaling Multi-task Pre-training with Task Prefix [PDF]. In this paper, we propose a task prefix guided multi-task pre-training framework (CompassMTL) to explore the relationships among tasks. CompassMTL is based on the DeBERTa architecture, trained with 40 natura 666B l language understanding tasks. Please refer more details in our paper.
- numpy
- torch
- transformers==4.17.0
- wandb
- sentencepiece
- sklearn
- datasets
Download data from datasets
Training:
bash run_train.shevaluate:
bash run_evaluate.shWe provide the models and outputs for the ANLI and HellaSwag Commonsense Reasoning tasks:
Our sinlge models for ANLI and HellaSwag are available at reasoning_models
The outputs can be found at model_outputs.
Please kindly cite this paper in your publications if it helps your research:
@inproceedings{zhang2022task,
title={Task Compass: Scaling Multi-task Pre-training with Task Prefix},
author={Zhang, Zhuosheng and Wang, Shuohang and Xu, Yichong and Fang, Yuwei and Yu, Wenhao and Liu, Yang and Zhao, Hai and Zhu, Chenguang and Zeng, Michael},
booktitle={arXiv preprint arXiv:2210.06277},
year={2022}
}