More Web Proxy on the site http://driver.im/

research-article

A Lightweight and Efficient Mechanism for Fixing the Synchronization of Misaligned Subtitle Documents

Authors:

Rodrigo Laiola Guimarães,

Priscilla Avegliano,

Lucas C. Villa RealAuthors Info & Claims

DocEng '16: Proceedings of the 2016 ACM Symposium on Document Engineering

Pages 175 - 184

https://doi.org/10.1145/2960811.2960812

Published: 13 September 2016 Publication History

Abstract

Online subtitle databases allow users to easily find subtitle documents in multiple languages for thousands of films and TV series episodes. However, getting the subtitle document that gives satisfactory synchronization on the first attempt is like hitting the jackpot. The truth is that this process often involves a lot of trial- and-error because multiple versions of subtitle documents have distinct synchronization references, given that they are targeted at variations of the same audiovisual content. Building on our previous efforts to address this problem, in this paper we formalize and validate a two-phase subtitle synchronization framework. The benefit over current approaches lays in the usage of audio fingerprint annotations generated from the base audio signal as second-level synchronization anchors. This way, we allow the media player to dynamically fix during playback the most common cases of subtitle synchronization misalignment that compromise users' watching experience. Results from our evaluation process indicate that our framework has minimal impact on existing subtitle documents and formats as well as on the playback performance.

References

[1]

Bartsch, M. A. and Wakefield, G. H. 2005. Audio thumbnailing of popular music using chroma-based representations. IEEE Transactions on Multimedia. 7, 1 (Feb. 2005), 96--104. DOI=http://dx.doi.org/10.1109/TMM.2004.840597

Digital Library

[2]

Brown, A., Jones, R., Crabb, M., Sandford, J., Brooks, M., Armstrong, M., and Jay, C. 2015. Dynamic Subtitles: The User Experience. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video (TVX '15). ACM, New York, NY, USA, 103--112. DOI=http://dx.doi.org/10.1145/2745197.2745204

Digital Library

[3]

Bulterman, D. C. A., Jansen, J, Cesar, P., and Cruz-Lara, S. 2007. An efficient, streamable text format for multimedia captions and subtitles. In Proceedings of the 2007 ACM symposium on Document engineering (DocEng '07). ACM, New York, NY, USA, 101--110. DOI=http://dx.doi.org/10.1145/1284420.1284451

Digital Library

[4]

Bulterman, D. C. A. and Rutledge, L. W. 2008. SMIL3.0 -- Flexible Multimedia for Web, Mobile Devices and DAISY Talking Books. (2nd ed.). Springer Publishing Company, Incorporated. ISBN: 978--3--540--78546-0

[5]

Cisco. 2015. Cisco Visual Networking Index: Forecast and Methodology, 2014--2019. White paper. Available at http://www.cisco.com/c/en/us/solutions/collateral/service-provider/ip-ngn-ip-next-generation-network/white_paper_c11--481360.pdf

[6]

Concolato, C. and Le Feuvre, J. 2013. Live HTTP streaming of video and subtitles within a browser. In Proceedings of the 4th ACM Multimedia Systems Conference (MMSys '13). ACM, New York, NY, USA, 146--150. DOI=http://dx.doi.org/10.1145/2483977.2483997

Digital Library

[7]

Fagá Jr., R., Motti, V. G., Cattelan, R. G., Teixeira, C. A. C., and Pimentel, M. G. C. 2010. A social approach to authoring media annotations. In Proceedings of the 10th ACM symposium on Document engineering (DocEng '10). ACM, New York, NY, USA, 17--26. DOI=http://dx.doi.org/10.1145/1860559.1860566

Digital Library

[8]

Federico, M. and Furini, M. 2014. An automatic caption alignment mechanism for off-the-shelf speech recognition technologies. Multimedia Tools Appl. 72, 1 (Sep. 2014), 21--40. DOI=http://dx.doi.org/10.1007/s11042-012--1318--3

Digital Library

[9]

Glickman, O., Dagan, I., Keller, M., Bengio, S., and Daelemans, W. 2006. Investigating lexical substitution scoring for subtitle generation. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X '06). Association for Computational Linguistics, Stroudsburg, PA, USA, 45--52.

Digital Library

[10]

Guimarães, R. L., Cesar, P., and Bulterman, D. C. A. 2010. Creating and sharing personalized time-based annotations of videos on the web. In Proceedings of the 10th ACM symposium on Document engineering (DocEng '10). ACM, New York, NY, USA, 27--36. DOI=http://dx.doi.org/10.1145/1860559.1860567

Digital Library

[11]

Hong, R., Wang, M. Yuan, X., Xu, M. Jiang, J., Yan, S., and Chua, T. 2011. Video accessibility enhancement for hearing-impaired users. ACM Trans. Multimedia Comput. Commun. Appl. 7S, 1, Article 24 (Nov. 2011), 19 pages. DOI=http://dx.doi.org/10.1145/2037676.2037681

Digital Library

[12]

Hughes, C. J., Armstrong, M., Jones, R., and Crabb, M. 2015. Responsive design for personalised subtitles. In Proceedings of the 12th Web for All Conference (W4A '15). ACM, New York, NY, USA, Article 8, 4 pages. DOI=http://dx.doi.org/10.1145/2745555.2746650

Digital Library

[13]

Kovacs, G. and Miller, R. C. 2014. Smart subtitles for vocabulary learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 853--862. DOI=http://dx.doi.org/10.1145/2556288.2557256

Digital Library

[14]

Kruger, J. L., Hefer, E., and Matthew, G. 2013. Measuring the impact of subtitles on cognitive load: eye tracking and dynamic audiovisual texts. In Proceedings of the 2013 Conference on Eye Tracking South Africa (ETSA '13). ACM, New York, NY, USA, 62--66. DOI=http://dx.doi.org/10.1145/2509315.2509331

Digital Library

[15]

Kushalnagar, R. S., Lasecki, W. S., and Bigham, J. P. 2013. Captions versus transcripts for online video content. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A '13). ACM, New York, NY, USA, Article 32, 4 pages. DOI=http://dx.doi.org/10.1145/2461121.2461142

Digital Library

[16]

Liu, X. and Wang, W. Robustly extracting captions in videos based on stroke-like edges and spatio-temporal analysis. IEEE Transactions on Multimedia. 14, 2 (Apr. 2012), 482--489. DOI=http://dx.doi.org/10.1109/TMM.2011.2177646

Digital Library

[17]

Rooney, K. 2014. The Impact of Keyword Caption Ratio on Foreign Language Listening Comprehension. International Journal of Computer-Assisted Language Learning and Teaching. 4, 2 (Apr. 2014), 11--28. DOI=http://dx.doi.org/10.4018/ijcallt.2014040102

Digital Library

[18]

Shimogori, N., Ikeda, T. and Tsuboi, S. 2010. Automatically generated captions: will they help non-native speakers communicate in english?. In Proceedings of the 3rd international conference on Intercultural collaboration (ICIC '10). ACM, New York, NY, USA, 79--86. DOI=http://dx.doi.org/10.1145/1841853.1841865

Digital Library

[19]

Tiedemann, J. 2008. Synchronizing translated movie subtitles. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC '08), 5 pages.

[20]

Villa Real, L. C., Guimarães, R. L., and Avegliano, P. 2015. Dynamic Adjustment of Subtitles Using Audio Fingerprints. In Proceedings of the 23rd ACM international conference on Multimedia (MM '15). ACM, New York, NY, USA, 975--978. DOI=http://dx.doi.org/10.1145/2733373.2806378

Digital Library

[21]

Wang, F., Nagano, H., Kashino, K., and Igarashi, T. 2015. Visualizing video sounds with sound word animation. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME '15). 1--6. DOI=http://dx.doi.org/10.1109/ICME.2015.7177422

Cited By

Weingärtner HWindl ML. Chuang LDraxler F(2024)Useful but Distracting: Viewer Experience with Keyword Highlights and Time-Synchronization in Captions for Language LearningProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701574(235-248)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3701574
Borges TGuimarães Rde Salles Soares Neto C(2020)Toward Assessing the Quality of Subtitles based on SRT Files ProcessingProceedings of the Brazilian Symposium on Multimedia and the Web10.1145/3428658.3431758(153-156)Online publication date: 30-Nov-2020
https://dl.acm.org/doi/10.1145/3428658.3431758
Guimarães RBrito JSantos CMota MSerique BPrates RCandello H(2018)Investigating the Influence of Subtitles Synchronization in the Viewer's Quality of ExperienceProceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems10.1145/3274192.3274222(1-10)Online publication date: 22-Oct-2018
https://dl.acm.org/doi/10.1145/3274192.3274222
Show More Cited By

Index Terms

A Lightweight and Efficient Mechanism for Fixing the Synchronization of Misaligned Subtitle Documents
1. Applied computing
  1. Document management and text processing
    1. Document management
      1. Document metadata
    2. Document preparation
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Speech / audio search
  2. Information systems applications
    1. Multimedia information systems

Recommendations

Dynamic Adjustment of Subtitles Using Audio Fingerprints
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

Anyone who ever downloaded subtitle files from the Internet has faced problems synchronizing them with the associated media files. Even with the efforts of communities on reviewing user-contributed subtitles and with mechanisms in movie players to ...
Investigating the Influence of Subtitles Synchronization in the Viewer's Quality of Experience
IHC '18: Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems

Watching a movie in which audio and video are out of sync is certainly an unpleasant experience, even with a big bucket of fresh popcorn. In a similar way, viewers can experience a negative experience when subtitles are not well synchronized with the ...
Toward understanding the quality of subtitle synchronization to improve the viewer experience
WebMedia '19: Proceedings of the 25th Brazillian Symposium on Multimedia and the Web

Whoever has already downloaded a subtitle file from the Internet has probably had a bitter disappointment when finding that the subtitles were not synchronized in regards to the audiovisual content. This situation demonstrates that the quality of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DocEng '16: Proceedings of the 2016 ACM Symposium on Document Engineering

September 2016

222 pages

ISBN:9781450344388

DOI:10.1145/2960811

General Chair:
Robert Sablatnig
TU Wien, Austria
,
Program Chair:
Tamir Hassan
HP Labs, Austria

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

In-Cooperation

SIGDOC: ACM Special Interest Group on Systems Documentation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

DocEng '16

Sponsor:

SIGWEB

DocEng '16: ACM Symposium on Document Engineering 2016

September 13 - 16, 2016

Vienna, Austria

Acceptance Rates

DocEng '16 Paper Acceptance Rate 11 of 35 submissions, 31%;

Overall Acceptance Rate 194 of 564 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
145
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Weingärtner HWindl ML. Chuang LDraxler F(2024)Useful but Distracting: Viewer Experience with Keyword Highlights and Time-Synchronization in Captions for Language LearningProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701574(235-248)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1145/3701571.3701574
Borges TGuimarães Rde Salles Soares Neto C(2020)Toward Assessing the Quality of Subtitles based on SRT Files ProcessingProceedings of the Brazilian Symposium on Multimedia and the Web10.1145/3428658.3431758(153-156)Online publication date: 30-Nov-2020
https://dl.acm.org/doi/10.1145/3428658.3431758
Guimarães RBrito JSantos CMota MSerique BPrates RCandello H(2018)Investigating the Influence of Subtitles Synchronization in the Viewer's Quality of ExperienceProceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems10.1145/3274192.3274222(1-10)Online publication date: 22-Oct-2018
https://dl.acm.org/doi/10.1145/3274192.3274222
Brito JGuimaraes RSantos CRoesler VValdeni de Lima JSaibel Santos CWillrich R(2017)Investigating the Collaborative Process of Subtitles Creation and Sharing for Videos on the WebProceedings of the 23rd Brazillian Symposium on Multimedia and the Web10.1145/3126858.3131592(69-72)Online publication date: 17-Oct-2017
https://dl.acm.org/doi/10.1145/3126858.3131592

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents