More Web Proxy on the site http://driver.im/

article

Applications of Geometry Processing: Visual storylines: Semantic visualization of movie sequence

Authors:

Shi-Min HuAuthors Info & Claims

Computers and Graphics, Volume 36, Issue 4

Pages 241 - 249

https://doi.org/10.1016/j.cag.2012.02.010

Published: 01 June 2012 Publication History

Abstract

This paper presents a video summarization approach that automatically extracts and visualizes movie storylines in a static image for the purposes of efficient representation and quick overview. A new type of video visualization, Visual Storylines, is designed to summarize video storylines in an image composition while preserving the style of the original videos. This is achieved with a series of video analysis, image synthesis, relationship quantification and geometric layout optimization techniques. Specifically, we analyze the video contents and quantify the video story unit relationships automatically through clustering video shots according to both the visual and audio data. A multi-level storyline visualization method then organizes and synthesizes a suitable amount of representative information, including both the locations and interested objects and characters, with the assistance of arrows, according to the relationships between the video story units and the temporal structure of the video sequence. Several results have demonstrated that our approach is able to abstract the main storylines of professionally edited video such as commercial movies and TV series, though some semantic key clues might be missed in the summarization. Preliminary user studies have been performed to evaluate our approach, and the results show that our approach can be used to assist viewers to understand video contents when they are familiar with the context of the video or when a text synopsis is provided.

References

[1]

Hua XS, Li S, Zhang HJ. Video booklet. In: ICME; 2005. p. 4-5.

[2]

Chiu P, Girgensohn A, Liu Q. Stained-glass visualization for highly condensed video summaries. In: IEEE international conference on multimedia and expo, vol. 3; 2004. p. 2059-62.

[3]

Yeung, M. and Yeo, B.L., Video visualization for compact presentation and fast browsing of pictorial content. IEEE Trans Circuits Syst Video Technol. v7 i5. 771-785.

[4]

Ma YF, Zhang HJ. Video snapshot: a bird view of video sequence. In: Proceedings of the 11th international multimedia modelling conference; 2005. p. 94-101.

[5]

Wang T, Mei T, Hua XS, Liu XL, Zhou HQ. Video collage: a novel presentation of video sequence. In: IEEE international conference on multimedia and expo; 2007. p. 1479-82.

[6]

Correa, C.D. and Ma, K.L., Dynamic video narratives. ACM Trans Graph. v29. 88-89.

[7]

Goldman, D.B., Curless, B., Seitz, S.M. and Salesin, D., Schematic storyboarding for video visualization and editing. ACM Trans Graph (Proc SIGGRAPH). v25 i3.

[8]

Money, A.G. and Agius, H., Video summarisation: a conceptual framework and survey of the state of the art. J Visual Commun Image Represent. v19 i2. 121-143.

[9]

Huet, B. and Merialdo, B., Automatic video summarization. In: Hammoud, R.I. (Ed.), Interactive video. Signals and communication technology, Springer, Berlin, Heidelberg. pp. 27-42.

[10]

Barnes, C., Goldman, D.B., Shechtman, E. and Finkelstein, A., Video tapestries with continuous temporal zoom. ACM Trans Graph. v29. 89-90.

[11]

Rui Y, Huang TS, Mehrotra S. Exploring video structure beyond the shots. In: Proceedings of IEEE conference on multimedia computing and systems; 1998. p. 237-40.

[12]

Yeung M, Yeo BL, Liu B. Extracting story units from long programs for video browsing and navigation. In: Proceedings of the third IEEE international conference on multimedia computing and systems; 1996. p. 296-305.

[13]

Ngo, C.W., Ma, Y.F. and Zhang, H.J., Video summarization and scene detection by graph modeling. IEEE Trans Circuits Syst Video Technol. v15 i2. 296-305.

[14]

Zhai Y, Shah M. Visual attention detection in video sequences using spatiotemporal cues. In: MULTIMEDIA '06: Proceedings of the 14th annual ACM international conference on multimedia. ACM, New York, NY, USA; 2006. p. 815-24. ISBN 1-59593-447-2.

Digital Library

[15]

Daniel G, Chen M. Video visualization. In: VIS '03: Proceedings of the 14th IEEE visualization 2003 (VIS'03). Washington, DC, USA: IEEE Computer Society; 2003. p. 54-5. ISBN 0-7695-2030-8.

Digital Library

[16]

Wang, Y., Liu, Z. and Huang, J.C., Multimedia content analysis-using both audio and visual clues. IEEE Signal Process Mag. v17 i6. 12-36.

[17]

Sugano M, Nakajima Y, Yanagihara H. Automated MPEG audio-video summarization and description. In: Proceedings of the IEEE international conference on image processing, vol. 1; 2002. p. I956-9.

[18]

He L, Sanocki E, Gupta A, Grudin J. Auto-summarization of audio-video presentations. In: 7th ACM international conference on multimedia; 1999. p. 489-98.

[19]

Lienhart, R.W., Comparison of automatic shot boundary detection algorithms. In: Yeung, M.M., Yeo, B.L., Bouman, C.A. (Eds.), Proceedings of the SPIE, storage and retrieval for image and video databases VII, presented at the society of photo-optical instrumentation engineers (SPIE) conference, vol. 3656. pp. 290-301.

[20]

Rabiner, L. and Schafer, R., Digital processing of speech signals. 1978. Prentice Hall, Englewood Cliffs.

[21]

Black, M.J. and Anandan, P., The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput Vis Image Underst. v63 i1. 75-104.

[22]

Lienhart R, Maydt J. An extended set of haar-like features for rapid object detection. In: IEEE ICIP 2002, vol. 1; 2002. p. 900-3.

[23]

Wang R, Huang T. Fast camera motion analysis in MPEG domain. In: ICIP 99. Proceedings of the 1999 International conference on Image processing, vol. 3; 1999. p. 691-4.

[24]

Cheng MM, Zhang GX, Mitra NJ, Huang X, Hu SM. Global contrast based salient region detection. In: IEEE CVPR; 2011. p. 409-16.

[25]

Rother C, Bordeaux L, Hamadi Y, Blake A. Autocollage. In: SIGGRAPH '06: ACM SIGGRAPH 2006 papers. ACM, New York, NY, USA; 2006. p. 847-52. ISBN 1-59593-364-6.

Digital Library

[26]

Huang, H., Zhang, L. and Zhang, H.C., Image collage: arcimboldo-like collage using internet images. ACM Trans Graph. v30 i6.

[27]

Teramoto, O., Park, I. and Igarashi, T., Interactive motion photography from a single image. Visual Comput. v26. 1339-1348.

[28]

Cheng, M.M., Zhang, F.L., Mitra, N.J., Huang, X. and Hu, S.M., Repfinder: finding approximately repeated scene elements for image editing. ACM Trans Graph. v29 i4. 83:1-83:8.

[29]

Fiss, J., Agarwala, A. and Curless, B., Candid portrait selection from video. ACM Trans Graph. v30 i6.

[30]

Chen, T., Cheng, M.M., Tan, P., Shamir, A. and Hu, S.M., Sketch2photo: internet image montage. ACM Trans Graph. v28 i5. 124:1-124:10.

[31]

Zhang, Y. and Tong, R., Environment-sensitive cloning in images. Visual Comput. v27. 739-748.

Cited By

Panzer LKurzhals KBeck F(2024)MoViQuotes: Creating Personalized Visual Quotes Capturing the Moods and Vibes of a MovieProceedings of Mensch und Computer 202410.1145/3670653.3677473(577-581)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3677473
Xia WSun GLi TChang BTang JZhang GLiang R(2024)Video Visualization and Visual Analytics: A Task-Based and Application- Driven InvestigationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.342340234:11_Part_2(11316-11339)Online publication date: 4-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3423402
Meena PKumar HYadav S(2024)A Volumetric Saliency Guided Image Summarization for RGB-D Indoor Scene ClassificationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.341294934:11_Part_1(10917-10929)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3412949
Show More Cited By

Applications of Geometry Processing: Visual storylines: Semantic visualization of movie sequence
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks

Recommendations

Interactive Exploration of Surveillance Video through Action Shot Summarization and Trajectory Visualization

We propose a novel video visual analytics system for interactive exploration of surveillance video data. Our approach consists of providing analysts with various views of information related to moving objects in a video. To do this we first extract each ...
Schematic storyboarding for video visualization and editing

We present a method for visualizing short video clips in a single static image, using the visual language of storyboards. These schematic storyboards are composed from multiple input frames and annotated using outlines, arrows, and text describing the ...
Schematic storyboarding for video visualization and editing
SIGGRAPH '06: ACM SIGGRAPH 2006 Papers

We present a method for visualizing short video clips in a single static image, using the visual language of storyboards. These schematic storyboards are composed from multiple input frames and annotated using outlines, arrows, and text describing the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computers and Graphics

Computers and Graphics Volume 36, Issue 4

June, 2012

91 pages

ISSN:0097-8493

Issue’s Table of Contents

Copyright © Elsevier Ltd © 2012.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 June 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Panzer LKurzhals KBeck F(2024)MoViQuotes: Creating Personalized Visual Quotes Capturing the Moods and Vibes of a MovieProceedings of Mensch und Computer 202410.1145/3670653.3677473(577-581)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3677473
Xia WSun GLi TChang BTang JZhang GLiang R(2024)Video Visualization and Visual Analytics: A Task-Based and Application- Driven InvestigationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.342340234:11_Part_2(11316-11339)Online publication date: 4-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3423402
Meena PKumar HYadav S(2024)A Volumetric Saliency Guided Image Summarization for RGB-D Indoor Scene ClassificationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.341294934:11_Part_1(10917-10929)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3412949
Meena PKumar HKumar Yadav S(2023)A review on video summarization techniquesEngineering Applications of Artificial Intelligence10.1016/j.engappai.2022.105667118:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.engappai.2022.105667
Sabha ASelwal A(2023)Data-driven enabled approaches for criteria-based video summarization: a comprehensive survey, taxonomy, and future directionsMultimedia Tools and Applications10.1007/s11042-023-14925-w82:21(32635-32709)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1007/s11042-023-14925-w
Tiwari VBhatnagar C(2021)A survey of recent work on video summarization: approaches and techniquesMultimedia Tools and Applications10.1007/s11042-021-10977-y80:18(27187-27221)Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.1007/s11042-021-10977-y
Basavarajaiah MSharma P(2019)Survey of Compressed Domain Video Summarization TechniquesACM Computing Surveys10.1145/335539852:6(1-29)Online publication date: 16-Oct-2019
https://dl.acm.org/doi/10.1145/3355398
Sun ZSun MCao NMa XChen WWeiskopf D(2016)VideoForestSIGGRAPH ASIA 2016 Symposium on Visualization10.1145/3002151.3002159(1-8)Online publication date: 28-Nov-2016
https://dl.acm.org/doi/10.1145/3002151.3002159
Tanahashi YChien-Hsin Hsueh Kwan-Liu Ma (2015)An Efficient Framework for Generating Storyline Visualizations from Streaming DataIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2015.239277121:6(730-742)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.1109/TVCG.2015.2392771
Jing GHu YGuo YYu YWang W(2015)Content-Aware Video2Comics With Manga-Style LayoutIEEE Transactions on Multimedia10.1109/TMM.2015.247426317:12(2122-2133)Online publication date: 1-Dec-2015
https://dl.acm.org/doi/10.1109/TMM.2015.2474263
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents