Abstract
Recent advancements in generative Large Language Models (LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A “Multi-Agent Text Evaluation framework" where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents’ interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.
Y. Li and S. Zhang—Equal Contributors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We have made the datasets and results used in our experiments publicly available at https://github.com/kse-ElEvEn/MATEval. Due to the user privacy of Alipay, we cannot make the “Ant" dataset public.
- 2.
This work was supported by Ant Group.
- 3.
This work was supported by the Natural Science Foundation of China (Grant No. U21A20488). We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.
References
Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL, pp. 65–72 (2005)
Callison-Burch, C.: Fast, cheap, and creative: evaluating translation quality using amazon’s mechanical turk. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, pp. 286–295 (2009)
Chan, C.M., et al: Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023)
Fu, J., et al: Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 (2023)
Ghazarian, S., et al: Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. arXiv preprint arXiv:1904.10635 (2019)
Guan, J., et al: Openmeva: a benchmark for evaluating open-ended story generation metrics. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), pp. 6394–6407 (2021)
Guan, J., et al: LOT: a story-centric benchmark for evaluating Chinese long text understanding and generation. Trans. Assoc. Comput . Lingust. 10, 434–451 (2022)
Guan, J., Huang, M.: UNION: an unreferenced metric for evaluating open-ended story generation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 9157–9166 (2020)
Li, G., et al: Camel: Communicative agents for“ mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760 (2023)
Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (Jul 2004)
Liu, C., et al: How to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 2122–2132 (2016)
Liu, Y., et al: G-eval: NLG evaluation using gpt-4 with better human alignment. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP. pp. 2511–2522 (2023)
Madaan, A., et al: Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 (2023)
Papineni, K., et al: Bleu: a method for automatic evaluation of machine translation. In: ACL, pp. 311–318 (2002)
Wang, J., et al: Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048 (2023)
Wang, P., et al: Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926 (2023)
Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural. Inf. Process. Syst. 35, 24824–24837 (2022)
Zhang, T., Kishore, V., Wu, F., Artzi, Y.: Bertscore: evaluating text generation with BERT. In: 8th International Conference on Learning Representations, ICLR (2020)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, Y. et al. (2024). MATEval: A Multi-agent Discussion Framework for Advancing Open-Ended Text Evaluation. In: Onizuka, M., et al. Database Systems for Advanced Applications. DASFAA 2024. Lecture Notes in Computer Science, vol 14856. Springer, Singapore. https://doi.org/10.1007/978-981-97-5575-2_31
Download citation
DOI: https://doi.org/10.1007/978-981-97-5575-2_31
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5574-5
Online ISBN: 978-981-97-5575-2
eBook Packages: Computer ScienceComputer Science (R0)