Abstract
The apparition of Large Language Models has attracted the interest of the research community as well as the general public due to the impressive improvement of their communicative and comprehensive capacities in general conversations. Nevertheless, there are still many domains that require further evaluation, especially those related to sensitive data and users, such as mental health. In this article, we evaluate several ensemble approaches to combine the Zero-Shot predictions of several families of open-source Language Models, specifically, RoBERTa and LLama-2, in the task of mental-health topics classification under limited data and computational resource conditions. With this purpose, we employed two datasets containing realistic questions and answers, Counsel-Chat and 7Cups datasets labeled in 28 and 39 fine-grain unbalanced mental-health topics. The best ensembles of non-fine-tuned models with Zero-Shot approaches achieved an accuracy (ACC) of 43.29%, weighted-F1 (W-F1) of 41.32% and Macro-F1 (M-F1) of 31.79% in the 28 topics of Counsel-Chat; and 35.57% of ACC, 39.66% W-F1 and 28.12% of M-F1 in the 39 topics of 7Cups dataset. The error analysis reveals that models have difficulties in detecting less concrete topics (e.g. ‘Social’), which suggests future lines to re-organize classes in topics and sub-topics, or the incorporation into the ensemble of models adapted to these domains to compensate for these errors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
‘nbertagnolli/counsel-chat’.
References
Anonymous: SBERT-InterprEVAL: SBERT for interpretability evaluation of transformer-derived architectures on mental-health topic classification (2024). Preprint at https://openreview.net/references/pdf?id=OuphQxftS9
Bertagnolli, N.: Counsel chat: bootstrapping high-quality therapy data (2020)
Brown, T., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901. Curran Associates, Inc. (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019, pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/N19-1423. https://aclanthology.org/N19-1423
Grootendorst, M.: BERTopic: neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022)
Hua, Y., et al.: Large language models in mental health care: a scoping review (2024)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach (2019)
World Health Organization: World mental health report: transforming mental health for all (2022). https://iris.who.int/bitstream/handle/10665/356119/9789240049338-eng.pdf?isAllowed=y &sequence=1
Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models (2023)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, pp. 6000–6010. Curran Associates Inc., Red Hook, NY, USA (2017)
Yang, K., Zhang, T., Kuang, Z., Xie, Q., Ananiadou, S.: MentaLLaMA: interpretable mental health analysis on social media with large language models. arXiv preprint arXiv:2309.13567 (2023)
Acknowledgments
The research leading to these results has received funding from the ‘CONVERSA: Effective and efficient resources and models for transformative conversational AI in Spanish and co-official languages’ project with reference TED2021-132470B-I00, funded by MCIN/AEI/10.13039/501100011033 and by the European Union “NextGenerationEU/PRTR”; European Union’s Horizon 2020 research and innovation program under grant agreement No. 823907 (MENHIR project: https://menhir-project.eu); and the Spanish R &D &i project GOMINOLA (PID2020-118112RB-C21 and PID2020-118112RB-C22) financed by MCIN/AEI/10.13039/501100011033.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Luna-Jiménez, C., Griol, D., Callejas, Z. (2024). Zero-Shot Ensemble of Language Models for Fine-Grain Mental-Health Topic Classification. In: Ferrández Vicente, J.M., Val Calvo, M., Adeli, H. (eds) Artificial Intelligence for Neuroscience and Emotional Systems. IWINAC 2024. Lecture Notes in Computer Science, vol 14674. Springer, Cham. https://doi.org/10.1007/978-3-031-61140-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-61140-7_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-61139-1
Online ISBN: 978-3-031-61140-7
eBook Packages: Computer ScienceComputer Science (R0)