Pattern Shifting or Knowledge Losing? A Forgetting Perspective for Understanding the Effect of Instruction Fine-Tuning
Authors: 
Chunkang Zhang, Boxi Cao, Yaojie Lu, Hongyu Lin, Liu Cao, Ke Zeng, Guanglu Wan, Xunliang Cai, Xianpei Han, Le SunAuthors Info & Claims
Pages 540 - 554
Abstract
Instruction Fine-Tuning (IFT) emerges as an essential step of training large language models to robustly carry out tasks of interest. However, there lacks a systematic investigation about the underlying mechanisms of instruction fine-tuning, particularly on the forgetting phenomenon after IFT, known as alignment tax. Therefore, to understand the mechanism of IFT from the forgetting perspective, we investigate the alternation of the text pattern and knowledge within models throughout the entire IFT process. Specifically, we restore fine-tuned models to their base version by training them on the data sharing a similar distribution with the pre-training corpus and compare their results Our experiment indicates that there is a stage transition of forgetting during IFT process: (1) Pseudo Forgetting: in this stage, models mainly shift their familiar text pattern away from pre-training data format while the world knowledge is preserved. Consequently, models will recover to their original performance when they are restored to the base version. (2) Actual Forgetting: in this stage, models forget the acquired knowledge as well. Therefore, they fail to reach the original performance even if they are restored to the base version.
References
[1]
Andersen, J.S., Maalej, W.: Efficient, uncertainty-based moderation of neural networks text classifiers. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 1536–1546. Association for Computational Linguistics, Dublin, Ireland (2022).
[2]
Bai, Y., et al.: Training a helpful and harmless assistant with reinforcement learning from human feedback (2022)
[3]
Burns, C., et al.: Weak-to-strong generalization: eliciting strong capabilities with weak supervision (2023)
[4]
Cha, H., Lee, J., Shin, J.: Co2l: contrastive continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9516–9525 (2021)
[5]
Cha, J., et al.: Swad: domain generalization by seeking flat minima (2021)
[6]
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding (2019)
[7]
Fischer G Lifelong learning-more than training J. Interact. Learn. Res. 2000 11 3 265-294
[8]
French RM Catastrophic forgetting in connectionist networks Trends Cogn. Sci. 1999 3 4 128-135
[9]
Goyal, S., Kumar, A., Garg, S., Kolter, Z., Raghunathan, A.: Finetune like you pretrain: improved finetuning of zero-shot vision models (2022)
[10]
Gupta, P., Jiao, C., Yeh, Y.T., Mehri, S., Eskenazi, M., Bigham, J.P.: Instructdial: improving zero and few-shot generalization in dialogue through instruction tuning (2022)
[11]
Hendrycks, D., et al.: Measuring massive multitask language understanding (2021)
[12]
Jain, S., et al.: Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (2023)
[13]
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017).
[14]
Kotha, S., Springer, J.M., Raghunathan, A.: Understanding Catastrophic Forgetting in Language Models via Implicit Inference (2023). http://arxiv.org/abs/2309.10105, arXiv:2309.10105 [cs]
[15]
Kumar, A., Raghunathan, A., Jones, R., Ma, T., Liang, P.: Fine-tuning can distort pretrained features and underperform out-of-distribution (2022)
[16]
Kung, P.N., Peng, N.: Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning (2023). http://arxiv.org/abs/2305.11383, arXiv:2305.11383 [cs]
[17]
Lin, Y., et al.: Speciality vs Generality: An Empirical Study on Catastrophic Forgetting in Fine-tuning Foundation Models (2023). http://arxiv.org/abs/2309.06256, arXiv:2309.06256 [cs]
[18]
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., Zhang, Y.: An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747 (2023)
[19]
McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev. 102(3), 419 (1995)
[20]
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation, vol. 24, pp. 109–165. Elsevier (1989)
[21]
Min, T.J.C.: An approach to solving the abstraction and reasoning corpus (arc) challenge (2023)
[22]
OpenAI Achiam, J., et al.: Gpt-4 technical report (2023)
[23]
Ouyang, L., et al.: Training language models to follow instructions with human feedback (2022)
[24]
Peng, B., Risteski, A.: Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions. Adv. Neural. Inf. Process. Syst. 35, 28414–28427 (2022)
[25]
Radford, A., et al.: Learning transferable visual models from natural language supervision (2021)
[26]
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: ICARL: incremental classifier and representation learning (2017)
[27]
Scialom, T., Chakrabarty, T., Muresan, S.: Fine-tuned language models are continual learners. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107–6122. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (2022)., https://aclanthology.org/2022.emnlp-main.410
[28]
Silver, D.L., Yang, Q., Li, L.: Lifelong machine learning systems: beyond learning algorithms. In: 2013 AAAI spring symposium series (2013)
[29]
Taori, R., et al.: Stanford alpaca: an instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca (2023)
[30]
Tirumala, K., Markosyan, A.H., Zettlemoyer, L., Aghajanyan, A.: Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022). http://arxiv.org/abs/2205.10770, arXiv:2205.10770 [cs]
[31]
Touvron, H., et al.: LLaMA: Open and Efficient Foundation Language Models (2023). http://arxiv.org/abs/2302.13971, arXiv:2302.13971 [cs]
[32]
Touvron, H., et al.: Llama 2: Open Foundation and Fine-Tuned Chat Models (2023). http://arxiv.org/abs/2307.09288, arXiv:2307.09288 [cs]
[33]
Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., Liu, T.: Huatuo: tuning llama model with Chinese medical knowledge (2023)
[34]
Wu, H., Tan, H., Xu, K., Liu, S., Wu, L., Song, L.: Zero-shot cross-lingual conversational semantic role labeling. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp. 269–281. Association for Computational Linguistics, Seattle, United States (2022).
[35]
Xia, M., et al.: Training trajectories of language models across scales (2023)
[36]
Yaddanapudi LN The American statistical association statement on p-values explained J. Anaesthesiol. Clin. Pharmacol. 2016 32 4 421
[37]
Yin, F., Vig, J., Laban, P., Joty, S., Xiong, C., Wu, C.S.J.: Did you read the instructions? Rethinking the effectiveness of task definitions in instruction learning (2023)
[38]
Zeng, S., et al.: Exploring memorization in fine-tuned language models (2023)
[39]
Zhang, M., Ré, C.: Contrastive adapters for foundation model group robustness (2022)
[40]
Zhao, Y., et al.: Pytorch fsdp: experiences on scaling fully sharded data parallel (2023)
[41]
Zheng, L., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena (2023)
Index Terms
- Pattern Shifting or Knowledge Losing? A Forgetting Perspective for Understanding the Effect of Instruction Fine-Tuning
Index terms have been assigned to the content through auto-classification.
Recommendations
Less-forgetting multi-lingual fine-tuning
NIPS '22: Proceedings of the 36th International Conference on Neural Information Processing SystemsMulti-lingual fine-tuning (MLF), which fine-tunes a multi-lingual language model (MLLM) with multiple source languages, aims to gain good zero-shot performance on target languages. In MLF, the fine-tuned model tends to fit the source languages while ...
Recall-Based Knowledge Distillation for Data Distribution Based Catastrophic Forgetting in Semantic Segmentation
Pattern RecognitionAbstractSemantic segmentation involves labeling each pixel in an image with a corresponding class label, enabling detailed scene understanding. In dynamic environments, where conditions change over time, incremental learning techniques are essential for ...
Understanding catastrophic forgetting for adaptive deep learning
CODS-COMAD '23: Proceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD)Deep learning is still limited in practice tho it has progressed state of the art over the past few years. Current deep learning algorithms are rigid and static once trained and can’t adapt to new data when deployed for inferencing. In this paper, we ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Jul 2024
603 pages
ISBN:978-981-97-8366-3
DOI:10.1007/978-981-97-8367-0
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 29 November 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 03 Feb 2025