More Web Proxy on the site http://driver.im/

research-article

CodingFlow: Enable Video Coding for Video Stabilization

Authors:

Shuaicheng Liu,

Bing ZengAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 26, Issue 7

Pages 3291 - 3302

https://doi.org/10.1109/TIP.2017.2697759

Published: 01 July 2017 Publication History

Abstract

Video coding focuses on reducing the data size of videos. Video stabilization targets at removing shaky camera motions. In this paper, we enable video coding for video stabilization by constructing the camera motions based on the motion vectors employed in the video coding. The existing stabilization methods rely heavily on image features for the recovery of camera motions. However, feature tracking is time-consuming and prone to errors. On the other hand, nearly all captured videos have been compressed before any further processing and such a compression has produced a rich set of block-based motion vectors that can be utilized for estimating the camera motion. More specifically, video stabilization requires camera motions between two adjacent frames. However, motion vectors extracted from video coding may refer to non-adjacent frames. We first show that these non-adjacent motions can be transformed into adjacent motions such that each coding block within a frame contains a motion vector referring to its adjacent previous frame. Then, we regularize these motion vectors to yield a spatially-smoothed motion field at each frame, named as CodingFlow, which is optimized for a spatially-variant motion compensation. Based on CodingFlow, we finally design a grid-based 2D method to accomplish the video stabilization. Our method is evaluated in terms of efficiency and stabilization quality, both quantitatively and qualitatively, which shows that our method can achieve high-quality results compared with the state-of-the-art methods (feature-based).

References

[1]

Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H.-Y. Shum, “ Full-frame video stabilization with motion inpainting,” IEEE Trans. Pattern Anal. Mach. Intell., vol. Volume 28, no. Issue 7, pp. 1150–1163, 2006.

Digital Library

[2]

F. Liu, M. Gleicher, H. Jin, and A. Agarwala, “ Content-preserving warps for 3D video stabilization,” ACM Trans. Graph., vol. Volume 28, no. Issue 3, 2009, Art. no. .

Digital Library

[3]

F. Liu, M. Gleicher, J. Wang, H. Jin, and A. Agarwala, “ Subspace video stabilization,” ACM Trans. Graph., vol. Volume 30, no. Issue 4, 2011, Art. no. .

Digital Library

[4]

B.-Y. Chen, K.-Y. Lee, W.-T. Huan, and J.-S. Lin, “ Capturing intention-based full-frame video stabilization,” Comput. Graph. Forum, vol. Volume 27, no. Issue 7, pp. 1805–1814, 2008.

[5]

S. Liu, L. Yuan, P. Tan, and J. Sun, “ Bundled camera paths for video stabilization,” ACM Trans. Graph., vol. Volume 32, no. Issue 4, 2013, Art. no. .

Digital Library

[6]

S. Liu, L. Yuan, P. Tan, and J. Sun, “ Steadyflow: Spatially smooth optical flow for video stabilization,” in Proc. CVPR, 2014, pp. 4209–4216.

Digital Library

[7]

M. Grundmann, V. Kwatra, and I. Essa, “ Auto-directed video stabilization with robust L1 optimal camera paths,” in Proc. CVPR, 2011, pp. 225–232.

Digital Library

[8]

M. Grundmann, V. Kwatra, D. Castro, and I. Essa, “ Calibration-free rolling shutter removal,” in Proc. ICCP, 2012, pp. 1–8.

[9]

Z. Zhou, H. Jin, and Y. Ma, “ Plane-based content-preserving warps for video stabilization,” in Proc. CVPR, 2013, pp. 2299–2306.

Digital Library

[10]

J. Shi and C. Tomasi, “ Good features to track,” in Proc. CVPR, 1994, pp. 593–600.

[11]

A. Goldstein and R. Fattal, “ Video stabilization using epipolar geometry,” ACM Trans. Graph., vol. Volume 32, no. Issue 5, 2012, Art. no. .

Digital Library

[12]

S. Bell, A. Troccoli, and K. Pulli, “ A non-linear filter for gyroscope-based video stabilization,” in Proc. ECCV, 2014, pp. 294–308.

[13]

A. Karpenko, D. E. Jacobs, J. Baek, and M. Levoy, “ Digital video stabilization and rolling shutter correction using gyroscopes,” <institution content-type=division>Stanford Comput. Sci</institution>., Stanford, CA, USA, Tech. Rep. CSTR 2011-03, 2011.

[14]

T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, “ Overview of the H.264/AVC video coding standard,” IEEE Trans. Circuits Syst. Video Technol., vol. Volume 13, no. Issue 7, pp. 560–576, 2003.

Digital Library

[15]

G. J. Sullivan, J. Ohm, W.-J. Han, and T. Wiegand, “ Overview of the high efficiency video coding (HEVC) standard,” IEEE Trans. Circuits Syst. Video Technol., vol. Volume 22, no. Issue 12, pp. 1649–1668, 2012.

Digital Library

[16]

M. L. Gleicher and F. Liu, “ Re-cinematography: Improving the camera dynamics of casual video,” in Proc. ACM Multimedia, 2007, pp. 27–36.

Digital Library

[17]

Y.-S. Wang, F. Liu, P.-S. Hsu, and T.-Y. Lee, “ Spatially and temporally optimized video stabilization,” IEEE Trans. Vis. Comput. Graphics, vol. Volume 19, no. Issue 8, pp. 1354–1361, 2013.

Digital Library

[18]

N. Ahmed, T. Natarajan, and K. R. Rao, “ Discrete cosine transform,” IEEE Trans. Comput., vol. Volume C-23, no. Issue 1, pp. 90–93, 1974.

Digital Library

[19]

B. Zeng and J. J. Fu, “ Directional discrete cosine transforms– A new framework for image coding,” IEEE Trans. Circuits Syst. Video Technol., vol. Volume 18, no. Issue 3, pp. 305–313, 2008.

Digital Library

[20]

SY. Zhu, S.-K. A. Yeung, and B. Zeng, “ R-D performance upper bound of transform coding for 2-D directional sources,” IEEE Signal Process. Lett., vol. Volume 16, no. Issue 10, pp. 861–864, 2009.

[21]

S. Y. Zhu, S.-K. A. Yeung, and B. Zeng, “ In search of “better-than-DCT” unitary transforms for encoding of residual signals,” IEEE Signal Process. Lett., vol. Volume 17, no. Issue 11, pp. 961–964, 2010.

[22]

P. K. Meher, S. Y. Park, B. K. Mohanty, K. S. Lim, and C. Yeo, “ Efficient integer DCT architectures for HEVC,” IEEE Trans. Circuits Syst. Video Technol., vol. Volume 24, no. Issue 1, pp. 168–178, 2014.

Digital Library

[23]

G. J. Sullivan and T. Wiegand, “ Rate-distortion optimization for video compression,” IEEE Signal Process. Mag., vol. Volume 15, no. Issue 6, pp. 74–90, 1998.

[24]

Y. Taki, M. Hatori, and S. Tanaka, “ Interframe coding that follows the motion,” in Proc. Inst. Electron. Commun. Eng. Jpn. Annu. Conv. (IECEJ), 1974, p. pp.1263.

[25]

J. Jain and A. Jain, “ Displacement measurement and its application in interframe image coding,” IEEE Trans. Commun., vol. Volume 29, no. Issue 12, pp. 1799–1808, 1981.

[26]

T. Koga, K. Linuma, A. Hirano, Y. Iijima, and T. Ishiguro, “ Motion-compensated interframe coding for video conferencing,” in Proc. NTC, 1981, pp. C9.6.1–C9.6.5.

[27]

R. Li, B. Zeng, and M. L. Liou, “ A new three-step search algorithm for block motion estimation,” IEEE Trans. Circuits Syst. Video Technol., vol. Volume 4, no. Issue 4, pp. 438–442, 1994.

Digital Library

[28]

L.-M. Po and W.-C. Ma, “ A novel four-step search algorithm for fast block motion estimation,” IEEE Trans. Circuits Syst. Video Technol., vol. Volume 6, no. Issue 3, pp. 313–317, 1996.

Digital Library

[29]

S. Zhu and K.-K. Ma, “ A new diamond search algorithm for fast block-matching motion estimation,” IEEE Trans. Image Process., vol. Volume 9, no. Issue 2, pp. 287–290, 2000.

Digital Library

[30]

S. Liu, Y. Wang, L. Yuan, J. Bu, P. Tan, and J. Sun, “ Video stabilization with a depth camera,” in Proc. CVPR, 2012, pp. 89–95.

Digital Library

[31]

B. M. Smith, L. Zhang, H. Jin, and A. Agarwala, “ Light field video stabilization,” in Proc. ICCV, 2009, pp. 341–348.

[32]

Z.-Q. Wang, L. Zhang, and H. Huang, “ Multiplane video stabilization,” Comput. Graph. Forum, vol. Volume 32, no. Issue 7, pp. 265–273, 2013.

[33]

C. Jia and B. L. Evans, “ Constrained 3D rotation smoothing via global manifold regression for video stabilization,” IEEE Trans. Signal Process., vol. Volume 62, no. Issue 13, pp. 3293–3304, 2014.

Digital Library

[34]

J. Bai, A. Agarwala, M. Agrawala, and R. Ramamoorthi, “ User-assisted video stabilization,” Comput. Graph. Forum, vol. Volume 33, no. Issue 4, pp. 61–70, 2014.

Digital Library

[35]

S. Liu, P. Tan, L. Yuan, J. Sun, and B. Zeng, “ Meshflow: Minimum latency online video stabilization,” in Proc. ECCV, 2016, pp. 800–815.

[36]

N. Jiang, P. Tan, and L.-F. Cheong, “ Multi-view repetitive structure detection,” in Proc. ICCV, 2011, pp. 535–542.

Digital Library

[37]

D. Glasner, S. Bagon, and M. Irani, “ Super-resolution from a single image,” in Proc. ICCV, 2009, pp. 349–356.

[38]

C. Barnes, E. Shechtman, A. Finkelstein, and D. Goldman, “ PatchMatch: A randomized correspondence algorithm for structural image editing,” ACM Trans. Graph., vol. Volume 28, no. Issue 3, p. pp.24, 2009.

Digital Library

[39]

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. New York, NY, USA: Cambridge Univ. Press, 2003.

Digital Library

[40]

D. Sun, S. Roth, and M. Black, “ Secrets of optical flow estimation and their principles,” in Proc. CVPR, 2010, pp. 2392–2399.

[41]

D. G. Lowe, “ Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. Volume 60, no. Issue 2, pp. 91–110, 2004.

Digital Library

Cited By

Liu SZhang ZLiu ZTan PZeng B(2025)Minimum Latency Deep Online Video Stabilization and Its ExtensionsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.349317547:2(1238-1249)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1109/TPAMI.2024.3493175
Liu YLi HLiu SZeng B(2024)CodingHomo: Bootstrapping Deep Homography With Video CodingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.341877134:11_Part_1(11214-11228)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3418771
Wang ZLiu XWang CJiang TZeng TZeng ZWang GLiu S(2024)Reconstruction flow recurrent network for compressed video quality enhancementPattern Recognition10.1016/j.patcog.2024.110638155:COnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.patcog.2024.110638
Show More Cited By

CodingFlow: Enable Video Coding for Video Stabilization
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
  2. Computer graphics
    1. Animation

Recommendations

True motion-compensated de-interlacing algorithm

This paper presents a true motion-compensated de-interlacing (TMCD) algorithm based on a fast true motion estimation scheme. The fast true motion estimation scheme, designated as a variable block size true motion estimation for a translated motion model ...
Lossy encoding of motion vectors using entropy-constrained vector quantization
ICIP '95: Proceedings of the 1995 International Conference on Image Processing (Vol. 3)-Volume 3 - Volume 3

Many well-known video coding algorithms employ block motion estimation and compensation. But not much attention has been given to the problem of rate allocation for encoding the motion vectors and motion compensated difference frame. We propose entropy-...
A Factorization-Based Approach for Articulated Nonrigid Shape, Motion and Kinematic Chain Recovery From Video

Recovering articulated shape and motion, especially human body motion, from video is a challenging problem with a wide range of applications in medical study, sport analysis and animation, etc. Previous work on articulated motion recovery generally ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 26, Issue 7

July 2017

522 pages

ISSN:1057-7149

Issue’s Table of Contents

Copyright © 2017.

Publisher

IEEE Press

Publication History

Published: 01 July 2017

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu SZhang ZLiu ZTan PZeng B(2025)Minimum Latency Deep Online Video Stabilization and Its ExtensionsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.349317547:2(1238-1249)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1109/TPAMI.2024.3493175
Liu YLi HLiu SZeng B(2024)CodingHomo: Bootstrapping Deep Homography With Video CodingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.341877134:11_Part_1(11214-11228)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3418771
Wang ZLiu XWang CJiang TZeng TZeng ZWang GLiu S(2024)Reconstruction flow recurrent network for compressed video quality enhancementPattern Recognition10.1016/j.patcog.2024.110638155:COnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.patcog.2024.110638
Wang NZhou CZhu RZhang BWang YLiu H(2024)SOFTEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107725130:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.engappai.2023.107725
Wang YHuang QJiang CLiu JShang MMiao Z(2023)Video stabilizationNeurocomputing10.1016/j.neucom.2022.10.008516:C(205-230)Online publication date: 7-Jan-2023
https://dl.acm.org/doi/10.1016/j.neucom.2022.10.008
Roberto e Souza MMaia HPedrini H(2022)Survey on Digital Video Stabilization: Concepts, Methods, and ChallengesACM Computing Surveys10.1145/349452555:3(1-37)Online publication date: 3-Feb-2022
https://dl.acm.org/doi/10.1145/3494525
Xu YZhang JMaybank STao D(2022)DUT: Learning Video Stabilization by Simply Watching Unstable VideosIEEE Transactions on Image Processing10.1109/TIP.2022.318288731(4306-4320)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1109/TIP.2022.3182887
Kumar RGupta SVenkatesh K(2022)Quadratic smoothing based video stabilization using spatio-temporal regularity flowMultimedia Tools and Applications10.1007/s11042-022-13570-z82:7(10337-10366)Online publication date: 26-Aug-2022
https://dl.acm.org/doi/10.1007/s11042-022-13570-z
Xie HXiao LWu H(2021)Intra- and Inter-frame Iterative Temporal Convolutional Networks for Video StabilizationProceedings of the 3rd ACM International Conference on Multimedia in Asia10.1145/3469877.3490608(1-7)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1145/3469877.3490608
Choi JKweon I(2020)Deep Iterative Frame Interpolation for Full-frame Video StabilizationACM Transactions on Graphics10.1145/336355039:1(1-9)Online publication date: 16-Jan-2020
https://dl.acm.org/doi/10.1145/3363550
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents