invited-talk

Deepfovea: neural reconstruction for foveated rendering and video compression using learned natural video statistics

Authors:

Gizem RufoAuthors Info & Claims

SIGGRAPH '19: ACM SIGGRAPH 2019 Talks

Article No.: 58, Pages 1 - 2

https://doi.org/10.1145/3306307.3328186

Published: 28 July 2019 Publication History

Get Access

Abstract

Recent advances in head-mounted displays (HMDs) provide new levels of immersion by delivering imagery straight to human eyes. The high spatial and temporal resolution requirements of these displays pose a tremendous challenge for real-time rendering and video compression. Since the eyes rapidly decrease in spatial acuity with increasing eccentricity, providing high resolution to peripheral vision is unnecessary. Upcoming VR displays provide real-time estimation of gaze, enabling gaze-contingent rendering and compression methods that take advantage of this acuity falloff. In this setting, special care must be given to avoid visible artifacts such as a loss of contrast or addition of flicker.

Supplementary Material

ZIP File (a58-kaplanyan.zip)

Supplemental material.

Download
10.58 MB

MP4 File (a58-kaplanyan.mp4)

Download
210.77 MB

MP4 File (gensub_306.mp4)

Download
215.33 MB

References

[1]

Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, 214--223.

Digital Library

Google Scholar

[2]

Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D Graphics. ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 6, Article 164 (2012), 164:1--164:10 pages.

Digital Library

Google Scholar

[3]

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral Normalization for Generative Adversarial Networks. CoRR abs/1802.05957 (2018).

Google Scholar

[4]

O. Ronneberger, P. Fischer, and T. Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI) (LNCS), Vol. 9351. 234--241.

Google Scholar

[5]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014).

Google Scholar

Cited By

View all

Li DDu RBabu ABrumar CVarshney A(2021)A Log-Rectilinear Transformation for Foveated 360-degree Video StreamingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.306776227:5(2638-2647)Online publication date: May-2021
https://doi.org/10.1109/TVCG.2021.3067762
Abrash M(2021)Creating the Future: Augmented Reality, the next Human-Machine Interface2021 IEEE International Electron Devices Meeting (IEDM)10.1109/IEDM19574.2021.9720526(1.2.1-1.2.11)Online publication date: 11-Dec-2021
https://doi.org/10.1109/IEDM19574.2021.9720526
Denes GJindal AMikhailiuk AMantiuk R(2020)A perceptual model of motion quality for rendering with adaptive refresh-rate and resolutionACM Transactions on Graphics10.1145/3386569.339241139:4(133:1-133:17)Online publication date: 12-Aug-2020
https://dl.acm.org/doi/10.1145/3386569.3392411
Show More Cited By

Recommendations

DeepFovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos

In order to provide an immersive visual experience, modern displays require head mounting, high image resolution, low latency, as well as high refresh rate. This poses a challenging computational problem. On the other hand, the human visual system can ...
Pipeline rendering: interaction and realism through hardware-based multi-pass rendering
Global illumination models for volume rendering

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGGRAPH '19: ACM SIGGRAPH 2019 Talks

July 2019

143 pages

ISBN:9781450363174

DOI:10.1145/3306307

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2019

Check for updates

Qualifiers

Invited-talk

Conference

SIGGRAPH '19

Sponsor:

SIGGRAPH

SIGGRAPH '19: Special Interest Group on Computer Graphics and Interactive Techniques Conference

July 28 - August 1, 2019

California, Los Angeles

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
453
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Li DDu RBabu ABrumar CVarshney A(2021)A Log-Rectilinear Transformation for Foveated 360-degree Video StreamingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.306776227:5(2638-2647)Online publication date: May-2021
https://doi.org/10.1109/TVCG.2021.3067762
Abrash M(2021)Creating the Future: Augmented Reality, the next Human-Machine Interface2021 IEEE International Electron Devices Meeting (IEDM)10.1109/IEDM19574.2021.9720526(1.2.1-1.2.11)Online publication date: 11-Dec-2021
https://doi.org/10.1109/IEDM19574.2021.9720526
Denes GJindal AMikhailiuk AMantiuk R(2020)A perceptual model of motion quality for rendering with adaptive refresh-rate and resolutionACM Transactions on Graphics10.1145/3386569.339241139:4(133:1-133:17)Online publication date: 12-Aug-2020
https://dl.acm.org/doi/10.1145/3386569.3392411
Meng XDu RVarshney A(2020)Eye-dominance-guided Foveated RenderingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.297344226:5(1972-1980)Online publication date: May-2020
https://doi.org/10.1109/TVCG.2020.2973442
Thumuluri VSharma M(2020)A Unified Deep Learning Approach for Foveated Rendering & Novel View Synthesis from Sparse RGB-D Light Fields2020 International Conference on 3D Immersion (IC3D)10.1109/IC3D51119.2020.9376340(1-8)Online publication date: 15-Dec-2020
https://doi.org/10.1109/IC3D51119.2020.9376340

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

DeepFovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos

Pipeline rendering: interaction and realism through hardware-based multi-pass rendering

Global illumination models for volume rendering