More Web Proxy on the site http://driver.im/

research-article

An object-based approach to image/video-based synthesis and processing for 3-D and multiview televisions

Authors:

Shing-Chow Chan,

Heung-Yeung ShumAuthors Info & Claims

IEEE Transactions on Circuits and Systems for Video Technology, Volume 19, Issue 6

Pages 821 - 831

Published: 01 June 2009 Publication History

Abstract

This paper proposes an object-based approach to a class of dynamic image-based representations called "plenoptic videos," where the plenoptic video sequences are segmented into image-based rendering (IBR) objects each with its image sequence, depth map, and other relevant information such as shape and alpha information. This allows desirable functionalities such as scalability of contents, error resilience, and interactivity with individual IBR objects to be supported. Moreover, the rendering quality in scenes with large depth variations can also be improved considerably. A portable capturing system consisting of two linear camera arrays was developed to verify the proposed approach. An important step in the object-based approach is to segment the objects in video streams into layers or IBR objects. To reduce the time for segmenting plenoptic videos under the semiautomatic technique, a new object tracking method based on the level-set method is proposed. Due to possible segmentation errors around object boundaries, natural matting with Bayesian approach is also incorporated into our system. Furthermore, extensions of conventional image processing algorithms to these IBR objects are studied and illustrated with examples. Experimental results are given to illustrate the efficiency of the tracking, matting, rendering, and processing algorithms under the proposed objectbased framework.

References

[1]

E. H. Adelson and J. Bergen, "The plenoptic function and the elements of early vision," in Comput. Models Visual Process., Cambridge, MA: MIT Press, 1991, pp. 3-20.

[2]

S. E. Chen, "QuickTime VR-An image-based approach to virtual environment navigation," in Proc. Comput. Graph. (SIGGRAPH'95), Aug. 1995, pp. 29-38.

[3]

M. Levoy and P. Hanrahan, "Light field rendering," in Proc. Comput. Graph. (SIGGRAPH'96), Aug. 1996, pp. 31-42.

[4]

S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," in Proc. Comput. Graph. (SIGGRAPH'96). Aug. 1996, pp. 43-54.

[5]

J. Shade, S. Gortler, L. W. He, and R. Szeliski, "Layered depth images," in Proc. Comput. Graph. SIGGRAPH'98, pp. 231-242.

[6]

H. Y. Shum, S. B. Kang, and S. C. Chan, "Survey of image-based representations and compression techniques," IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 11, pp. 1020-1037, Nov. 2003.

Digital Library

[7]

H. Y. Shum, S. C. Chan, and S. B. Kang, Image-Based Rendering, New York: Springer-Verlag, 2006.

[8]

J. X. Chai, X. Tong, S. C. Chan, and H. Y. Shum, "Plenoptic sampling," in Proc. Comput. Graph. (SIGGRAPH'OO), Jul. 2000, pp. 307-318.

[9]

H. Y. Shum, J. Sun, S. Yamazaki, Y. Li, and C. K. Tang, "Pop-up light field: An interactive image-based modeling and rendering system," ACM Trans. Graph., vol. 23, no. 2, pp. 143-162, Apr. 2004.

Digital Library

[10]

K. T. Ng, S. C. Chan, and H. Y. Shum, "The data compression and transmission aspects of panoramic videos," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 1, pp. 82-95, Jan. 2005.

Digital Library

[11]

Z. F. Gan, S. C. Chan, K. T. Ng, K. L. Chan, and H. Y. Shum, "On the rendering and post-processing of simplified dynamic light fields with depth information," in Proc. IEEE ICASSP, Vol. 3, May 2004, pp. 321-324.

[12]

S. C. Chan, K. T. Ng, Z. F. Gan, K. L. Chan, and H. Y. Shum, "The compression of simplified dynamic light fields," in Proc. IEEE ICASSP, vol. 3, Apr. 2003, pp. 653-656.

[13]

S. C. Chan, K. T. Ng, Z. F. Gan, K. L. Chan, and H Y. Shum, "The plenoptic videos: Capturing, rendering and compression," in Proc. IEEE ISCAS, vol. 3, May 2004, pp. 905-908.

[14]

S. C. Chan, K. T. Ng, Z. F. Gan, K. L. Chan, and H. Y. Shum, "The plenoptic videos," IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 12, pp. 1650-1659, Dec. 2005.

Digital Library

[15]

Z. F. Gan, S. C. Chan, K. T. Ng, and H. Y. Shum, "An object-based approach to plenoptic videos," in Proc. IEEE Int. Symp. Circuits Syst., vol. 4, May 2005, pp. 3435-3438.

[16]

Z. F. Gan, S. C. Chan, K. T. Ng, and H. Y. Shum, "Object tracking for a class of dynamic image-based representations," in Proc. SPIE Visual Commun. Image Process., Jul. 2005, pp. 1267-1274.

[17]

Z. F. Gan. S. C. Chan, and H. Y. Shum, "Object tracking and matting for a class of dynamic image-based representations," in Proc. IEEE Advanced Video Signal-Based Surveillance, Sep. 2005, pp. 81-86.

[18]

Y. Li, J. Sun, C. K. Tang, and H. Y. Shum, "Lazy snapping," in Proc. Comput. Graph. (SIGGRAPH'04), pp. 303-308.

[19]

T. F. Chan and L. A. Vese, "Active contours without edges," IEEE Trans. Image Process., vol. 10, no. 2, pp. 266-277, Feb. 2001.

Digital Library

[20]

Y. Y. Chuang, B. Curless, D. Salesin, and R. Szeliski, "A bayesian approach to digital matting," in Proc. IEEE Conf. CVPR, vol. 2, Dec. 2001, pp. 264-271.

[21]

R. C. Gonzalez and R. E. Woods, Digital Image Processing. Englewood Cliffs, NJ: Prentice Hall, 2002.

[22]

B. Wilburn. M. Smulski, H. H. Lee, and M. Horowitz, "The light field video camera," in Proc. SPIE Electron. Imaging: Media Process. '2002, vol. 4674, Jan. 2002, pp. 29-36.

[23]

T. Naemura, J. Tago, and H. Harashima, "Real-time videobased modeling and rendering of 3-D scenes," IEEE Trans. Comput. Graph. Applicat., vol. 22, no. 2, pp. 66-73, Mar.-Apr. 2002.

Digital Library

[24]

J. C. Yang, M. Everett, C. Buehler, and L. McMillan, "A real-time distributed light field camera," in Proc. Eurograph. Workshop Rendering, 2002, pp. 77-86.

[25]

B. Goldlücke, M. Magnor, and B. Wilburn, "Hardware-accelerated dynamic light field rendering," in Proc. VMV'2002, pp. 455-462.

[26]

Z. Zhang, "A flexible new technique for camera calibration," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 11, pp. 1330-1334, Nov. 2000.

Digital Library

[27]

A. R. Mansouri, "Region tracking via level set PDEs without motion computation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 947-961, Jul. 2002.

Digital Library

[28]

N. Paragios and R. Deriche, "Geodesic active contours and levels sets for the detection and tracking of moving objects," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 3, pp. 266-280, Mar. 2000.

Digital Library

[29]

G. Sapiro, Geometric Partial Differential Equations and Image Analysis. Cambridge, U.K.: Cambridge Univ. Press, 2001.

Digital Library

[30]

J. A. Sethian, Level Set Methods: Evolving Interfaces in Geometry, Fluid Mechanics, Computer Vision and Materials Sciences. Cambridge, u.K.: Cambridge Univ. Press, 1996.

[31]

S. J. Osher and R. P. Fedkiw, Level Set Methods and Dynamic Implicit Surfaces. New York: Springer-Verlag, 2002.

[32]

S. J. Osher and J. A. Sethian, "Fronts propagation with curvature dependent speed: Algorithms based on Hamilton-Jacobi formulations," J. Comput. Phys., vol. 79, no. 1, pp. 12-49, 1988.

Digital Library

[33]

A. Yilmaz, X. Li, and M. Shah, "Object contour tracking using level sets," in Proc. ACCV 2004, Korea.

[34]

K. J. Yoon and I. S. Kweon, "Adaptive support-weight approach for correspondence search," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 650-656, Apr. 2006.

Digital Library

[35]

S. S. Intille and A. F. Bobick, "Disparity-space images and large occlusion stereo," in Proc. ECCV, 1994, pp. 179-186.

[36]

T. Porter and T. Duff, "Compositing digital image," in Proc. Comput. Graph. (SIGGRAPH'84), Jul. 1984, pp. 253-259.

[37]

J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani, "Hierarchical model-based motion estimation," in Proc. Eur. Conf. Comput. Vision, Santa Margharita Ligure, Italy, 1992, pp. 237-252.

Cited By

Chan SZhu ZNg KWang CZhang SZhang ZYe ZShum H(2012)The Design and Construction of a Movable Image-Based Rendering System and Its Application to Multiview ConferencingJournal of Signal Processing Systems10.1007/s11265-010-0566-667:3(305-316)Online publication date: 1-Jun-2012
https://dl.acm.org/doi/10.1007/s11265-010-0566-6
Ishii MTakahashi KNaemura T(2010)Joint rendering and segmentation of free-viewpoint videoJournal on Image and Video Processing10.1155/2010/7639202010(1-12)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.1155/2010/763920

An object-based approach to image/video-based synthesis and processing for 3-D and multiview televisions
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

A GPU favor representation method for plenoptic-illumination function based on an efficient spherical partition scheme

In image-based relighting (IBR), a set of reference images is pre-captured under various light directions. These reference images represent the appearance of a scene with an adjustable illumination. This paper describes a graphics process unit (GPU) ...
Object-Based Coding for Plenoptic Videos

A new object-based coding system for a class of dynamic image-based representations called plenoptic videos (PVs) is proposed. PVs are simplified dynamic light fields, where the videos are taken at regularly spaced locations along line segments instead ...
Photometric Image-Based Rendering for Virtual Lighting Image Synthesis
IWAR '99: Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality

A concept named Photometric Image-Based Rendering (PIBR) is introduced for a seamless augmented reality. The PIBR is defined as Image-Based Rendering which covers appearance changes caused by the lighting condition changes, while Geometric Image-Based ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Circuits and Systems for Video Technology

IEEE Transactions on Circuits and Systems for Video Technology Volume 19, Issue 6

June 2009

136 pages

ISSN:1051-8215

Issue’s Table of Contents

Copyright © 2009.

Publisher

IEEE Press

Publication History

Published: 01 June 2009

Revised: 30 June 2007

Received: 06 January 2007

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chan SZhu ZNg KWang CZhang SZhang ZYe ZShum H(2012)The Design and Construction of a Movable Image-Based Rendering System and Its Application to Multiview ConferencingJournal of Signal Processing Systems10.1007/s11265-010-0566-667:3(305-316)Online publication date: 1-Jun-2012
https://dl.acm.org/doi/10.1007/s11265-010-0566-6
Ishii MTakahashi KNaemura T(2010)Joint rendering and segmentation of free-viewpoint videoJournal on Image and Video Processing10.1155/2010/7639202010(1-12)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.1155/2010/763920

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents