More Web Proxy on the site http://driver.im/

research-article

Video retargeting with multi-scale trajectory optimization

Authors:

Wen GaoAuthors Info & Claims

MIR '10: Proceedings of the international conference on Multimedia information retrieval

Pages 45 - 54

https://doi.org/10.1145/1743384.1743399

Published: 29 March 2010 Publication History

Abstract

Mobile devices are increasingly powerful in media storage and rendering. The prevalent request of decent video browsing on mobile devices is demanding. However, one limitation comes from the size and aspect constraints of display. To display a video on a small screen, rendering process probably undergoes a sort of retargeting to fit into the target display and keep the most of original video information. In this paper, we formulate video retargeting as the problem of finding an optimal trajectory for a cropping window to go through the video, capturing the most salient region to scale towards proper display on the target. To measure the visual importance of every pixel, we utilize the local spatial-temporal saliency (ST-saliency) and face detection results. The spatiotemporal movement of the cropping window is modeled in a graph where a smoothed trajectory is resolved by a Max-Flow/Min-Cut method in a global optimization manner. Based on the horizontal/vertical projections and a graph-based method, the trajectory estimation of each shot can be conducted within one second. Also, the process of merging trajectories is employed to capture more saliency in video. Experimental results on diverse video contents have shown that our approach is efficient and subjective evaluation shows that the retargeted video has gained desirable user satisfaction.

References

[1]

F. Liu and M. Gleicher. Video retargeting: automating pan and scan. In Proceedings of the ACM international conference on Multimedia, pages. 241--250, 2006.

Digital Library

[2]

L. Wolf, M. Guttmann, D. Cohen-Or. Non-homogeneous content-driven video-retargeting. In Proceedings of the IEEE 11th International Conference on Computer Vision, pages 1 -- 6, 2007.

[3]

W. H. Chen, C. W. Wang, J. L. Wu. Video Adaptation for small display based on content recomposition. IEEE Transactions on Circuits and Systems for Video Technology, 17(1), pages 43--58, 2007.

Digital Library

[4]

L. Itti and C. Koch. Computational modeling of visual attention. Nature Rev. Neurosci., 2(3), pages 194--203, Mar. 2001.

[5]

L. Itti, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11), pages 1254--1259, Nov. 1998.

Digital Library

[6]

L. Itti and P. Baldi. A principled approach to detecting surprising events in video. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 631--637, 2005.

Digital Library

[7]

P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie. Behavior recognition via sparse spatio-temporal features. In Proceedings of International Conference on Computer Communications and Networks, pages 65--72, 2005.

Digital Library

[8]

B. Suh, H. Ling, B. B. Bederson, and D. W. Jacobs. Automatic thumbnail cropping and its effectiveness. In Proceedings of the 16th annual ACM symposium on User interface software and technology, pages 95--104, 2003.

Digital Library

[9]

X. Xie, H. Liu, W.-Y. Ma, and H.-J. Zhang. Browsing Large Pictures Under Limited Display Sizes. IEEE Transactions on Multimedia, 8(4), pages 707--715, 2006.

Digital Library

[10]

X. Fan, X. Xie, H.-Q. Zhou, and W.-Y. Ma. Looking into video frames on small displays. In Proceedings of the ACM international conference on Multimedia, pages 247--250, 2003.

Digital Library

[11]

J. Wang, M. J. Reinders, R. L. Lagendijk, J. Lindenberg, and M. S. Kankanhalli. Video content representation on tiny devices. In Proceedings of the IEEE International Conference on Multimedia and Expo, pages 1711--1714, 2004.

[12]

E. A. Hazem, J. David, D. Larry. Multi-scale video cropping. In Proceedings of the ACM international conference on Multimedia, pages 97--106, 2007.

Digital Library

[13]

J. Chen, R. Wang, S. Yan, S. Shan, X. Chen, Wen Gao. Face detection based on the example resampling by manifold. IEEE Transactions on System Man, and Cybernetics (Part A), 37(6), pages 1017--1028, Nov. 2007.

Digital Library

[14]

S. Engel, X. Zhang, and B. Wandell. Color tuning in human visual cortex measured with functional magnetic resonance imaging. Nature, 388 (6637), pages 68--71, 1997.

[15]

C.-W. Ngo, T.-C. Pong, and H.-J. Zhang. Motion analysis and segmentation through spatio-temporal slices processing. IEEE Transactions on Image Process, 12(3), pages 341--355, 2003.

Digital Library

[16]

G. Granlund and H. Knutsson. Signal processing for computer vision. Kluwer Academic Publishers, Dordrecht, Netherlands, 1995.

Digital Library

[17]

M. R. Lyu, J, Song, and M. Cai. A comprehensive method for multilingual video text detection, localization, and extraction. IEEE Transactions on Circuits and Systems for Video Technology, 5(2), pages 243--255, 2005.

Digital Library

[18]

T. H. Cormen, C. E. Leiserson, and R. L. Rivest. Introduction to Algorithms. McGraw-Hill, New York, 1990

Digital Library

[19]

Y. Boykov, V. Kolmogorov. An experimental comparison of Min-Cut/Max-Flow algorithm for energe minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(9), pages 1224--1237, Sept. 2004.

Digital Library

[20]

A. Torralba and R. Fergus and W. T. Freeman. Tiny image. In MIT-CSAIL-TR-2007-024.

[21]

D. Gao, M. Vijay and V. Nuno. Discriminant center-surround hypothesis for bottom-up saliency, In Proceedings of Neural Information Processing Systems (NIPS), 2007.

[22]

L. Shi, J. Wang, L. Y. Duan, H. Lu. Consumer Video Retargeting: Context Assisted Spatial-temporal Grid Optimization. In Proceedings of the ACM international conference on Multimedia, pages 301--310, 2009.

Digital Library

[23]

D. Thomas, D. Philippe, and N. Hermann. Pan, zoom, scan time-coherent, trained automatic video cropping. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, pages 1--8, 2008.

[24]

M. Rubinstein, A. Shamir, and S. Avidan. Improved seam carving for video retargeting. In Proceedings of ACM International Conference on SIGGRAPH, 2008.

Digital Library

[25]

J. Li, Y. Tian, T. Huang and W. Gao, A dataset and evaluation methodology for visual saliency in video, in Proceedings of IEEE International Conference on Multimedia and Expo, 2009.

Digital Library

Cited By

Tang ZLv CTang Y(2022)Adaptive cropping with interframe relative displacement constraint for video retargetingSignal Processing: Image Communication10.1016/j.image.2022.116666104(116666)Online publication date: May-2022
https://doi.org/10.1016/j.image.2022.116666
Kiess JKopf SGuthier BEffelsberg W(2018)A Survey on Content-Aware Image and Video RetargetingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/323159814:3(1-28)Online publication date: 24-Jul-2018
https://dl.acm.org/doi/10.1145/3231598
Du HXu ZYan Z(2016)Video Retargeting for Intelligent Sensing of Surveillance DevicesFrontier Computing10.1007/978-981-10-0539-8_119(1173-1178)Online publication date: 20-Apr-2016
https://doi.org/10.1007/978-981-10-0539-8_119
Show More Cited By

Index Terms

Video retargeting with multi-scale trajectory optimization
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

Video retargeting: automating pan and scan
MM '06: Proceedings of the 14th ACM international conference on Multimedia

When a video is displayed on a smaller display than originally intended, some of the information in the video is necessarily lost. In this paper, we introduce Video Retargeting that adapts video to better suit the target display, minimizing the ...
Motion-based video retargeting with optimized crop-and-warp
SIGGRAPH '10: ACM SIGGRAPH 2010 papers

We introduce a video retargeting method that achieves high-quality resizing to arbitrary aspect ratios for complex videos containing diverse camera and dynamic motions. Previous content-aware retargeting methods mostly concentrated on spatial ...
Motion-aware temporal coherence for video resizing
SIGGRAPH Asia '09: ACM SIGGRAPH Asia 2009 papers

Temporal coherence is crucial in content-aware video retargeting. To date, this problem has been addressed by constraining temporally adjacent pixels to be transformed coherently. However, due to the motion-oblivious nature of this simple constraint, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MIR '10: Proceedings of the international conference on Multimedia information retrieval

March 2010

600 pages

ISBN:9781605588155

DOI:10.1145/1743384

General Chairs:
James Z. Wang
The Pennsylvania State University, USA
,
Nozha Boujemaa
INRIA, France
,
Program Chairs:
Nuria Oliver Ramirez
Telefonica Research, Spain
,
Apostol Natsev
IBM Research, USA

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 March 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MIR '10

Sponsor:

SIGMM

MIR '10: International Conference on Multimedia Information Retrieval

March 29 - 31, 2010

Pennsylvania, Philadelphia, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
327
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tang ZLv CTang Y(2022)Adaptive cropping with interframe relative displacement constraint for video retargetingSignal Processing: Image Communication10.1016/j.image.2022.116666104(116666)Online publication date: May-2022
https://doi.org/10.1016/j.image.2022.116666
Kiess JKopf SGuthier BEffelsberg W(2018)A Survey on Content-Aware Image and Video RetargetingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/323159814:3(1-28)Online publication date: 24-Jul-2018
https://dl.acm.org/doi/10.1145/3231598
Du HXu ZYan Z(2016)Video Retargeting for Intelligent Sensing of Surveillance DevicesFrontier Computing10.1007/978-981-10-0539-8_119(1173-1178)Online publication date: 20-Apr-2016
https://doi.org/10.1007/978-981-10-0539-8_119
Lin XMa YMa LZhang R(2014)A survey for image resizingJournal of Zhejiang University SCIENCE C10.1631/jzus.C140010215:9(697-716)Online publication date: 23-Sep-2014
https://doi.org/10.1631/jzus.C1400102
Kiess JGritzner DGuthier BKopf SEffelsberg WZimmermann R(2014)GPU video retargeting with parallelized SeamCropProceedings of the 5th ACM Multimedia Systems Conference10.1145/2557642.2557648(139-147)Online publication date: 19-Mar-2014
https://dl.acm.org/doi/10.1145/2557642.2557648
Du HLiu ZYan ZJiang Y(2014)Video Retargeting Based on Stretchability-Aware Blocks ScalingProceedings of the 2014 10th International Conference on Semantics, Knowledge and Grids10.1109/SKG.2014.9(171-176)Online publication date: 27-Aug-2014
https://dl.acm.org/doi/10.1109/SKG.2014.9
Du HLiu ZWang JMei LHe Y(2014)Video Retargeting Based on Spatiotemporal Saliency ModelFuture Information Technology10.1007/978-3-642-55038-6_61(397-402)Online publication date: 2014
https://doi.org/10.1007/978-3-642-55038-6_61
Li JGao WLi JGao W(2014)Conclusions and Future WorkVisual Saliency Computation10.1007/978-3-319-05642-5_9(233-237)Online publication date: 2014
https://doi.org/10.1007/978-3-319-05642-5_9
Li JGao WLi JGao W(2014)Saliency-Based ApplicationsVisual Saliency Computation10.1007/978-3-319-05642-5_8(215-232)Online publication date: 2014
https://doi.org/10.1007/978-3-319-05642-5_8
Yo WLeou JHsiao H(2013)Video retargeting using non-homogeneous scaling and cropping2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference10.1109/APSIPA.2013.6694167(1-5)Online publication date: Oct-2013
https://doi.org/10.1109/APSIPA.2013.6694167
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents