Article

Free access

Entropy rate constancy in text

Authors:

Dmitriy Genzel,

Eugene CharniakAuthors Info & Claims

ACL '02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics

Pages 199 - 206

https://doi.org/10.3115/1073083.1073117

Published: 06 July 2002 Publication History

PDF eReader

Abstract

We present a constancy rate principle governing language generation. We show that this principle implies that local measures of entropy (ignoring context) should increase with the sentence number. We demonstrate that this is indeed the case by measuring entropy in three different ways. We also show that this effect has both lexical (which words are used) and non-lexical (how the words are used) causes.

References

[1]

M. P. Aylett. 1999. Stochastic suprasegmentals: Relationships between redundancy, prosodic structure and syllabic duration. In Proceedings of ICPhS--99, San Francisco.

Google Scholar

[2]

E. Charniak. 2001. A maximum-entropy-inspired parser. In Proceedings of ACL--2001, Toulouse.

Digital Library

Google Scholar

[3]

J. T. Goodman. 2001. A bit of progress in language modeling. Computer Speech and Language, 15:403--434.

Digital Library

Google Scholar

[4]

I. Kontoyiannis, P. H. Algoet, Yu. M. Suhov, and A. J. Wyner. 1998. Nonparametric entropy estimation for stationary processes and random fields, with applications to English text. IEEE Trans. Inform. Theory, 44:1319--1327, May.

Digital Library

Google Scholar

[5]

I. Kontoyiannis. 1996. The complexity and entropy of literary styles. NSF Technical Report No. 97, Department of Statistics, Stanford University, June. {unpublished, can be found at the author's web page}.

Google Scholar

[6]

R. Kuhn and R. De Mori. 1990. A cache-based natural language model for speech reproduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):570--583.

Digital Library

Google Scholar

[7]

M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn treebank. Computational Linguistics, 19:313--330.

Digital Library

Google Scholar

[8]

J. B. Plotkin and M. A. Nowak. 2000. Language evolution and information theory. Journal of Theoretical Biology, pages 147--159.

Crossref

Google Scholar

[9]

C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27:379--423, 623--656, July, October.

Crossref

Google Scholar

Cited By

View all

Wu GQian JCastelo Quispe SChen SRulff JSilva C(2024)ARTiST: Automated Text Simplification for Task Guidance in Augmented RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642772(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642772
Yang ZYuan YXu YZhan SBai HChen KOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)FACEProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666867(17038-17056)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666867
Liu JNikolaus MBodur KFourtassi A(2022)Predicting Backchannel Signaling in Child-Caregiver Multimodal ConversationsCompanion Publication of the 2022 International Conference on Multimodal Interaction10.1145/3536220.3563372(196-200)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536220.3563372
Show More Cited By

Entropy rate constancy in text
1. Computing methodologies
  1. Artificial intelligence
2. Hardware
  1. Power and energy
    1. Power estimation and optimization

Recommendations

Some properties of Rényi entropy and Rényi entropy rate

In this paper, we define the conditional Renyi entropy and show that the so-called chain rule holds for the Renyi entropy. Then, we introduce a relation for the rate of Renyi entropy and use it to derive the rate of the Renyi entropy for an irreducible-...
Rényi entropy rate for Gaussian processes

In this paper, we introduce the definition of the conditional Renyi entropy for continuous random variables and show that the so-called chain rule holds. Then, we use this rule to obtain another relation for getting the rate of Renyi entropy. Using this ...
Text analysis and language identification for polyglot text-to-speech synthesis

In multilingual countries, text-to-speech synthesis systems often have to deal with texts containing inclusions of multiple other languages in form of phrases, words, or even parts of words. In such multilingual cultural settings, listeners expect a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ACL '02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics

July 2002

543 pages

General Chair:
Pierre Isabelle

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 06 July 2002

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 85 of 443 submissions, 19%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
761
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)9

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wu GQian JCastelo Quispe SChen SRulff JSilva C(2024)ARTiST: Automated Text Simplification for Task Guidance in Augmented RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642772(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642772
Yang ZYuan YXu YZhan SBai HChen KOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)FACEProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3666867(17038-17056)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3666867
Liu JNikolaus MBodur KFourtassi A(2022)Predicting Backchannel Signaling in Child-Caregiver Multimodal ConversationsCompanion Publication of the 2022 International Conference on Multimodal Interaction10.1145/3536220.3563372(196-200)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536220.3563372
Braverman MChen XKakade SNarasimhan KZhang CZhang YDaumé HSingh A(2020)Calibration, entropy rates, and memory in language modelsProceedings of the 37th International Conference on Machine Learning10.5555/3524938.3525040(1089-1099)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.5555/3524938.3525040
Lara-Alvarez CAlvarado-Hernandez MMitre-Hernandez HGonzález-Calleros J(2017)Automatic evaluation of learning objects based on cross-entropy of eye fixations minimizationProceedings of the XVIII International Conference on Human Computer Interaction10.1145/3123818.3123872(1-4)Online publication date: 25-Sep-2017
https://dl.acm.org/doi/10.1145/3123818.3123872
Dethlefs NHastie HCuayáhuitl HYu YRieser VLemon O(2016)Information density and overlap in spoken dialogueComputer Speech and Language10.1016/j.csl.2015.11.00137:C(82-97)Online publication date: 1-May-2016
https://dl.acm.org/doi/10.1016/j.csl.2015.11.001
Leiva LSanchis-Trilles GJones MPalanque PSchmidt AGrossman T(2014)Representatively memorableProceedings of the SIGCHI Conference on Human Factors in Computing Systems10.1145/2556288.2557024(1709-1712)Online publication date: 26-Apr-2014
https://dl.acm.org/doi/10.1145/2556288.2557024
Islam ZRahman MMehler A(2014)Readability Classification of Bangla TextsProceedings of the 15th International Conference on Computational Linguistics and Intelligent Text Processing - Volume 840410.1007/978-3-642-54903-8_42(507-518)Online publication date: 6-Apr-2014
https://dl.acm.org/doi/10.1007/978-3-642-54903-8_42
Dethlefs NHastie HRieser VLemon OTsujii JHenderson JPasca M(2012)Optimising incremental dialogue decisions using information density for interactive systemsProceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning10.5555/2390948.2390959(82-93)Online publication date: 12-Jul-2012
https://dl.acm.org/doi/10.5555/2390948.2390959
Maurits LPerfors ANavarro D(2010)Why are some word orders more common than others? A uniform information density accountProceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 210.5555/2997046.2997073(1585-1593)Online publication date: 6-Dec-2010
https://dl.acm.org/doi/10.5555/2997046.2997073
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Recommendations

Some properties of Rényi entropy and Rényi entropy rate

Rényi entropy rate for Gaussian processes

Text analysis and language identification for polyglot text-to-speech synthesis

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations