More Web Proxy on the site http://driver.im/

Article

Free access

A comparison of rankings produced by summarization evaluation measures

Authors:

Robert L. Donaway,

Kevin W. Drummey,

Laura A. MatherAuthors Info & Claims

NAACL-ANLP-AutoSum '00: Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4

Pages 69 - 78

https://doi.org/10.3115/1117575.1117583

Published: 30 April 2000 Publication History

Abstract

Summary evaluation measures produce a ranking of all possible extract summaries of a document. Recall-based evaluation measures, which depend on costly human-generated ground truth summaries, produce uncorrelated rankings when ground truth is varied. This paper proposes using sentence-rank-based and content-based measures for evaluating extract summaries, and compares these with recall-based evaluation measures. Content-based measures increase the correlation of rankings induced by synonymous ground truths, and exhibit other desirable properties.

References

[1]

Ronald Brandow, Karl Mitze, and Lisa F. Rau. 1995. Automatic condensation of electronic publications by sentence selection. Information Processing and Management, 31(5):675--685.

Digital Library

[2]

Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391--407.

[3]

Susan T. Dumais. 1991. Improving the retrieval of information from external sources. Behavior Research Methods, Instruments & Computers, 23(2):229--236.

[4]

Therese Firmin and Michael J. Chrzanowski. 1999. An evaluation of automatic text summarization systems. In Advances in Automatic Text Summarization, chapter 21, pages 325--336. MIT Press, Cambridge, Massachusetts.

[5]

Jade Goldstein, Mark Kantrowitz, Vibhu Mittal, and Jaime Carbonell. 1999. Summarizing text documents: Sentence selection and evaluation metrics. In Proceedings of the ACM SIGIR, pages 121--128.

Digital Library

[6]

Gene H. Golub and Charles F. van Loan. 1989. Matrix Computations. The Johns Hopkins University Press, Baltimore.

[7]

Hongyan Jing, Kathleen McKeown, Regina Barzilay, and Michael Elhadad. 1998. Summarization evaluation methods: Experiments and analysis. In American Association for Artificial Intelligence Spring Symposium Series, pages 60--68.

[8]

Julian Kupiec, Jan Pederson, and Francine Chen. 1995. A trainable document summarizer. In Proceedings of the ACM SIGIR, pages 68--73.

Digital Library

[9]

Inderjeet Mani, David House, Gary Klein, Lynette Hirschman, Therese Firmin, and Beth Sundheim. 1999. The TIPSTER SUMMAC text summarization evaluation. In Proceedings of the Ninth Conference of the European Chapter of the ACL, pages 77--85.

Digital Library

[10]

Daniel Marcu. 1997. From discourse structure to text summaries. In Proceedings of the ACL'97/EACL'97 Workshop on Intelligent Scalable Text Summarization, pages 82--88.

[11]

C. D. Paice and P. A. Jones. 1993. The identification of important concepts in highly structured technical papers. In Proceedings of the ACM SIGIR, pages 69--78.

Digital Library

[12]

Gerard Salton. 1989. Automatic Text Processing. Addison-Wesley Publishers, Massachusetts.

Digital Library

[13]

C. Schwarz. 1990. Content based text handling. Information Processing and Management, 26(2):219--226.

Digital Library

[14]

David J. Sheskin. 1997. Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press LLC, United States.

[15]

T. Strzalkowski, J. Wang, and B. Wise. 1998. A robust practical text summarizaton system. In AAAI Intelligent Text Summarization Workshop, pages 26--30.

[16]

Ellen M. Voorhees. 1998. Variations in relevance judgements and the measurement of retrieval effectiveness. In Proceedings of the ACM SIGIR, pages 315--323.

Digital Library

Cited By

Chen PWu FWang TDing WMcIlraith SWeinberger K(2018)A semantic QA-based approach for text summarization evaluationProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence10.5555/3504035.3504623(4800-4807)Online publication date: 2-Feb-2018
https://dl.acm.org/doi/10.5555/3504035.3504623
Boudia MHamou RAmine A(2018)Comparative Study Between Two Swarm Intelligence Automatic Text SummariesInternational Journal of Applied Metaheuristic Computing10.4018/IJAMC.20180101029:1(15-39)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.4018/IJAMC.2018010102
Antunes JLins RLima ROliveira HRiss MSimske S(2018)Automatic cohesive summarization with pronominal anaphora resolutionComputer Speech and Language10.1016/j.csl.2018.05.00452:C(141-164)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1016/j.csl.2018.05.004
Show More Cited By

Recommendations

A comparison of rankings produced by summarization evaluation measures
NAACL-ANLP-AutoSum '00: Proceedings of the 2000 NAACL-ANLP Workshop on Automatic Summarization

Summary evaluation measures produce a ranking of all possible extract summaries of a document. Recall-based evaluation measures, which depend on costly human-generated ground truth summaries, produce uncorrelated rankings when ground truth is varied. ...
The heterogeneity principle in evaluation measures for automatic summarization
NAACL HLT '12: Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization

The development of summarization systems requires reliable similarity (evaluation) measures that compare system outputs with human references. A reliable measure should have correspondence with human judgements. However, the reliability of measures ...
Properties of Group Fairness Measures for Rankings
In recent years, several measures have been developed for evaluating group fairness of rankings. Given that these measures were developed with different application contexts and ranking algorithms in mind, it is not straightforward which measure to choose ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image DL Hosted proceedings

NAACL-ANLP-AutoSum '00: Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic summarization - Volume 4

April 2000

112 pages

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 30 April 2000

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 21 of 29 submissions, 72%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
541
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)7

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen PWu FWang TDing WMcIlraith SWeinberger K(2018)A semantic QA-based approach for text summarization evaluationProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence10.5555/3504035.3504623(4800-4807)Online publication date: 2-Feb-2018
https://dl.acm.org/doi/10.5555/3504035.3504623
Boudia MHamou RAmine A(2018)Comparative Study Between Two Swarm Intelligence Automatic Text SummariesInternational Journal of Applied Metaheuristic Computing10.4018/IJAMC.20180101029:1(15-39)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.4018/IJAMC.2018010102
Antunes JLins RLima ROliveira HRiss MSimske S(2018)Automatic cohesive summarization with pronominal anaphora resolutionComputer Speech and Language10.1016/j.csl.2018.05.00452:C(141-164)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1016/j.csl.2018.05.004
Lokbani A(2017)A New Metric of Validation for Automatic Text Summarization by ExtractionInternational Journal of Strategic Information Technology and Applications10.4018/IJSITA.20170701028:3(20-40)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.4018/IJSITA.2017070102
Troullinou GKondylakis HDaskalaki EPlexousakis DGandon FSabou MSack H(2017)Ontology understanding without tearsSemantic Web10.3233/SW-1702648:6(797-815)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.3233/SW-170264
Wang WLi ZWang JZheng Z(2017)How far we can go with extractive text summarization? Heuristic methods to obtain near upper boundsExpert Systems with Applications: An International Journal10.1016/j.eswa.2017.08.04090:C(439-463)Online publication date: 30-Dec-2017
https://dl.acm.org/doi/10.1016/j.eswa.2017.08.040
Hamou RAmine ABoudia MRahmani A(2015)A New Biomimetic Method Based on the Power Saves of Social Bees for Automatic Summaries of Texts by ExtractionInternational Journal of Software Science and Computational Intelligence10.4018/IJSSCI.20150101027:1(18-38)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.4018/IJSSCI.2015010102
Saggion HTorres-Moreno JCunha ISanJuan EJoshi AHuang CJurafsky D(2010)Multilingual summarization evaluation without human modelsProceedings of the 23rd International Conference on Computational Linguistics: Posters10.5555/1944566.1944688(1059-1067)Online publication date: 23-Aug-2010
https://dl.acm.org/doi/10.5555/1944566.1944688
Ceylan HMihalcea RÖzertem ULloret EPalomar MKaplan R(2010)Quantifying the limits and success of extractive summarization systems across domainsHuman Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics10.5555/1857999.1858132(903-911)Online publication date: 2-Jun-2010
https://dl.acm.org/doi/10.5555/1857999.1858132
Louis ANenkova AKoehn PMihalcea R(2009)Automatically evaluating content selection in summarization without human modelsProceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 110.5555/1699510.1699550(306-314)Online publication date: 6-Aug-2009
https://dl.acm.org/doi/10.5555/1699510.1699550
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents