More Web Proxy on the site http://driver.im/

research-article

Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency

Authors:

Xu SunAuthors Info & Claims

ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), Volume 18, Issue 3

Article No.: 31, Pages 1 - 15

https://doi.org/10.1145/3314934

Published: 30 April 2019 Publication History

Abstract

Abstractive text summarization is a highly difficult problem, and the sequence-to-sequence model has shown success in improving the performance on the task. However, the generated summaries are often inconsistent with the source content in semantics. In such cases, when generating summaries, the model selects semantically unrelated words with respect to the source content as the most probable output. The problem can be attributed to heuristically constructed training data, where summaries can be unrelated to the source content, thus containing semantically unrelated words and spurious word correspondence. In this article, we propose a regularization approach for the sequence-to-sequence model and make use of what the model has learned to regularize the learning objective to alleviate the effect of the problem. In addition, we propose a practical human evaluation method to address the problem that the existing automatic evaluation method does not evaluate the semantic consistency with the source content properly. Experimental results demonstrate the effectiveness of the proposed approach, which outperforms almost all the existing models. Especially, the proposed approach improves the semantic consistency by 4% in terms of human evaluation.

References

[1]

Armen Aghajanyan. 2017. SoftTarget regularization: An effective technique to reduce over-fitting in neural networks. In Proceedings of the 3rd IEEE International Conference on Cybernetics. IEEE, New York, NY, 1--5.

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2014), 15. arxiv:1409.0473

[3]

Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI Press, Palo Alto, California, 4784--4791. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16121

[4]

Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang. 2016. Distraction-based neural networks for modeling documents. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. AAAI Press/International Joint Conferences on Artificial Intelligence, Palo Alto, California, 2754--2760. http://www.ijcai.org/Abstract/16/391

Digital Library

[5]

Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. The Association for Computational Linguistics, Stroudsburg, Pennsylvania, 93--98.

[6]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 1 (1960), 37--46.

[7]

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers. The Association for Computer Linguistics, Stroudsburg, PA, 1631--1640.

[8]

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR abs/1503.02531 (2015), 9. arxiv:1503.02531

[9]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735--1780.

Digital Library

[10]

Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. LCSTS: A large scale Chinese short text summarization dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. The Association for Computer Linguistics, Stroudsburg, PA, 1967--1972.

[11]

Sébastien Jean, KyungHyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers. Association for Computer Linguistics, Stroudsburg, PA, 1--10.

[12]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014), 15. arxiv:1412.6980

[13]

J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biom. 33, 1 (1977), 159--174.

[14]

Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the 27th International Conference on Computational Linguistics. ACL, Stroudsburg, PA, 1430--1441. http://aclweb.org/anthology/C18-1121

[15]

Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. 2017. Deep recurrent generative decoder for abstractive text summarization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. The Association for Computer Linguistics, Stroudsburg, PA, 2081--2090.

[16]

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 74--81. http://aclweb.org/anthology/W04-1013

[17]

Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. 2018. Global encoding for abstractive summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 2: Short Papers. The Association for Computer Linguistics, Stroudsburg, PA, 163--169. http://aclweb.org/anthology/P18-2027

[18]

Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. The Association for Computer Linguistics, Stroudsburg, PA, 1412--1421.

[19]

Shuming Ma, Xu Sun, Wei Li, Sujian Li, Wenjie Li, and Xuancheng Ren. 2018. Query and output: Generating words by querying distributed word representations for paraphrase generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1: Long Papers. The Association for Computer Linguistics, Stroudsburg, PA, 196--206.

[20]

Shuming Ma, Xu Sun, Junyang Lin, and Xuancheng Ren. 2018. A hierarchical end-to-end model for jointly improving text summarization and sentiment classification. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence, Palo Alto, California, 4251--4257.

Digital Library

[21]

Shuming Ma, Xu Sun, Jingjing Xu, Houfeng Wang, Wenjie Li, and Qi Su. 2017. Improving semantic relevance for sequence-to-sequence learning of Chinese social media text summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 2: Short Papers. The Association for Computer Linguistics, Stroudsburg, PA, 635--640.

[22]

Julian John McAuley and Jure Leskovec. 2013. From amateurs to connoisseurs: Modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd International World Wide Web Conference. International World Wide Web Conferences Steering Committee. ACM, Geneva, Switzerland, New York, NY. 897--908.

Digital Library

[23]

Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. The Association for Computer Linguistics, Stroudsburg, PA, 2283--2288.

[24]

Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016. The Association for Computer Linguistics, Stroudsburg, PA, 280--290.

[25]

Quang Nguyen, Hamed Valizadegan, and Milos Hauskrecht. 2014. Learning classification models with soft-label information. J. Am. Med. Inf. Assoc. 21, 3 (2014), 501--508.

[26]

Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. The Association for Computer Linguistics, Stroudsburg, PA, 379--389.

[27]

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers. The Association for Computer Linguistics, Stroudsburg, PA, 1073--1083.

[28]

Kaiqiang Song, Lin Zhao, and Fei Liu. 2018. Structure-infused copy mechanisms for abstractive summarization. In Proceedings of the 27th International Conference on Computational Linguistics. The Association for Computer Linguistics, Stroudsburg, PA, 1717--1729. http://aclweb.org/anthology/C18-1146

[29]

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014. Curran Associates, Inc., Red Hook, NY, 3104--3112. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks

Digital Library

Cited By

Li QWan WZhao YJiang X(2024)Improved BIO-Based Chinese Automatic Abstract-Generation ModelACM Transactions on Asian and Low-Resource Language Information Processing10.1145/364369523:3(1-16)Online publication date: 9-Mar-2024
https://dl.acm.org/doi/10.1145/3643695
Cao KCheng WHao YGan YGao RZhu JWu J(2024)DMSeqNet-mBART: A state-of-the-art Adaptive-DropMessage enhanced mBART architecture for superior Chinese short news text summarizationExpert Systems with Applications10.1016/j.eswa.2024.125095257(125095)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.125095
Ghanem FPadma MAlkhatib R(2023)Automatic Short Text Summarization Techniques in Social Media PlatformsFuture Internet10.3390/fi1509031115:9(311)Online publication date: 13-Sep-2023
https://doi.org/10.3390/fi15090311
Show More Cited By

Index Terms

Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Natural language generation
  2. Machine learning
    1. Machine learning algorithms
      1. Regularization
    2. Machine learning approaches
      1. Neural networks

Recommendations

Single-Document Abstractive Text Summarization: A Systematic Literature Review
Abstractive text summarization is a task in natural language processing that automatically generates the summary from the source document in a human-written form with minimal loss of information. Research in text summarization has shifted towards ...
Abstractive text summarization and new large-scale datasets for agglutinative languages Turkish and Hungarian
Abstract
Due to the exponential growth in the number of documents on the Web, accessing the salient information relevant to a user need is gaining importance, which increases the popularity of text summarization. Recent progress in deep learning shifted ...
Abstractive text summarization using LSTM-CNN based deep learning

Abstractive Text Summarization (ATS), which is the task of constructing summary sentences by merging facts from different source sentences and condensing them into a shorter representation while preserving information content and overall meaning. It is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian and Low-Resource Language Information Processing

ACM Transactions on Asian and Low-Resource Language Information Processing Volume 18, Issue 3

September 2019

386 pages

ISSN:2375-4699

EISSN:2375-4702

DOI:10.1145/3305347

Editor:
Nianwen Xue
Brandeis University, Waltham, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 April 2019

Accepted: 01 January 2019

Revised: 01 September 2018

Received: 01 May 2018

Published in TALLIP Volume 18, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
212
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li QWan WZhao YJiang X(2024)Improved BIO-Based Chinese Automatic Abstract-Generation ModelACM Transactions on Asian and Low-Resource Language Information Processing10.1145/364369523:3(1-16)Online publication date: 9-Mar-2024
https://dl.acm.org/doi/10.1145/3643695
Cao KCheng WHao YGan YGao RZhu JWu J(2024)DMSeqNet-mBART: A state-of-the-art Adaptive-DropMessage enhanced mBART architecture for superior Chinese short news text summarizationExpert Systems with Applications10.1016/j.eswa.2024.125095257(125095)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.125095
Ghanem FPadma MAlkhatib R(2023)Automatic Short Text Summarization Techniques in Social Media PlatformsFuture Internet10.3390/fi1509031115:9(311)Online publication date: 13-Sep-2023
https://doi.org/10.3390/fi15090311
Yu FZhang PDing XLu TGu N(2023)BNoteHelper: A Note-based Outline Generation Tool for Structured Learning on Video-sharing PlatformsACM Transactions on the Web10.1145/363877518:2(1-30)Online publication date: 27-Dec-2023
https://dl.acm.org/doi/10.1145/3638775
Zhang MZhou GYu WHuang NLiu W(2023)GA-SCS: Graph-Augmented Source Code SummarizationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/355482022:2(1-19)Online publication date: 21-Feb-2023
https://dl.acm.org/doi/10.1145/3554820
Kavitha JKiran JPrasad SSoma KBabu GSivakumar S(2022)Prediction and Its Impact on Its Attributes While Biasing MachineLearning Training Data2022 Third International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE)10.1109/ICSTCEE56972.2022.10100010(1-7)Online publication date: 16-Dec-2022
https://doi.org/10.1109/ICSTCEE56972.2022.10100010
Yadala SGurram AS YU VLatha SChitti T(2022)Social Media Event Summarization using Neural Networks2022 Third International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE)10.1109/ICSTCEE56972.2022.10099963(1-7)Online publication date: 16-Dec-2022
https://doi.org/10.1109/ICSTCEE56972.2022.10099963
Xi XPi ZZhou G(2020)Global Encoding for Long Chinese Text SummarizationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/340791119:6(1-17)Online publication date: 6-Oct-2020
https://dl.acm.org/doi/10.1145/3407911

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents