More Web Proxy on the site http://driver.im/

article

Deep Shading: Convolutional Neural Networks for Screen Space Shading

Authors:

E. Arabadzhiyska,

T. RitschelAuthors Info & Claims

Computer Graphics Forum, Volume 36, Issue 4

Pages 65 - 78

https://doi.org/10.1111/cgf.13225

Published: 01 July 2017 Publication History

Abstract

In computer vision, convolutional neural networks CNNs achieve unprecedented performance for inverse problems where RGB pixel appearance is mapped to attributes such as positions, normals or reflectance. In computer graphics, screen space shading has boosted the quality of real-time rendering, converting the same kind of attributes of a virtual scene back to appearance, enabling effects like ambient occlusion, indirect light, scattering and many more. In this paper we consider the diagonal problem: synthesizing appearance from given per-pixel attributes using a CNN. The resulting Deep Shading renders screen space effects at competitive quality and speed while not being programmed by human experts but learned from example images.

References

[1]

<label>{BSD08}</label> Bavoil L., Sainz M., Dimitrov R.: Image-space horizon-based ambient occlusion. In ACM SIGGRAPH 2008 Talks2008. 6

Digital Library

[2]

<label>{BZM07}</label> Bosch A., Zisserman A., Munoz X.: Image classification using random forests and ferns. In Proc. ICCV2007, pp. pp.1-8. 13

[3]

<label>{CB15}</label> Christensen P.H., Burley B.: Approximate Reflectance Profiles for Efficient Subsurface Scattering. Tech. rep., 2015. 4, 8

[4]

<label>{CMK*14}</label> Cimpoi M., Maji S., Kokkinos I., Mohamed S., Vedaldi A.: Describing textures in the wild. In CVPR2014. 5

[5]

<label>{CS13}</label> Criminisi A., Shotton J.: Decision forests for computer vision and medical image analysis. Springer Science & Business Media, 2013. 13

Digital Library

[6]

<label>{Dac11}</label> Dachsbacher C.: Analyzing visibility configurations. IEEE Trans. Vis. and Comp. Graph. Volume 17, Issue 14 2011, pp.475-86. 2

Digital Library

[7]

<label>{DTSB15}</label> Dosovitskiy A., Tobias Springenberg J., Brox T.: Learning to generate chairs with convolutional neural networks. In Proc. CVPR2015, pp. pp.1538-1546. 3

[8]

<label>{EPF14}</label> Eigen D., Puhrsch C., Fergus R.: Depth map prediction from a single image using a multi-scale deep network. In Proc. NIPS2014, pp. pp.2366-74. 2

Digital Library

[9]

<label>{ERS13}</label> Elek O., Ritschel T., Seidel H.-P.: Real-time screen-space scattering in homogeneous environments. IEEE Computer Graph. and App., Volume 3 2013, pp.53-65. 2

Digital Library

[10]

<label>{FFL11}</label> Farbman Z., Fattal R., Lischinski D.: Convolution pyramids. ACM Trans. Graph. Proc. SIGGRAPH Volume 30, Issue 6 2011, pp.175:1-175:8. 2

Digital Library

[11]

<label>{GDDM14}</label> Girshick R., Donahue J., Darrell T., Malik J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. CVPR2014, pp. pp.580-7. 2

Digital Library

[12]

<label>{GEB15}</label> Gatys L.A., Ecker A.S., Bethge M.: A neural algorithm of artistic style. arXiv1508.06576 2015. 2, 9, 10

[13]

<label>{HAGM15}</label> Hariharan B., Arbeláez P., Girshick R., Malik J.: Hypercolumns for object segmentation and fine-grained localization. In Proc. CVPR2015. 3

[14]

<label>{Has86}</label> Hastad J.: Almost optimal lower bounds for small depth circuits. In ACM Theory of computing 1986, ACM, pp. pp.6-20. 13

Digital Library

[15]

<label>{Her03}</label> Hertzmann A.: Machine learning for computer graphics: A manifesto and tutorial. In Proc. Pacific Graphics2003. 2

Digital Library

[16]

<label>{JDA*11}</label> Johnson M.K., Dale K., Avidan S., Pfister H., Freeman W.T., Matusik W.: CG2Real: Improving the realism of computer generated images using a large collection of photographs. IEE Trans. Vis. and Comp. Graph. Volume 17, Issue 9 2011, pp.1273-85. 2

Digital Library

[17]

<label>{JSD*14}</label> Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., Darrell T.: Caffe: Convolutional architecture for fast feature embedding. In Proc. ACM Multimedia2014, pp. pp.675-8. 5

Digital Library

[18]

<label>{JSG09}</label> Jimenez J., Sundstedt V., Gutierrez D.: Screen-space perceptual rendering of human skin. ACM Trans. Applied Perception Volume 6, Issue 4 2009, pp.23. 2, 8

Digital Library

[19]

<label>{Kaj86}</label> Kajiya J.T.: The rendering equation. In ACM SIGGRAPH 1986, vol. Volume 20, pp. pp.143-50. 2

Digital Library

[20]

<label>{KBS15}</label> Kalantari N.K., Bako S., Sen P.: A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. Proc. SIGGRAPH2015. 2, 13

Digital Library

[21]

<label>{KSH12}</label> Krizhevsky A., Sutskever I., Hinton G.E.: Imagenet classification with deep convolutional neural networks. In Proc. NIPS2012, pp. pp.1097-105. 2

Digital Library

[22]

<label>{KWKT15}</label> Kulkarni T.D., Whitney W., Kohli P., Tenenbaum J.B.: Deep convolutional inverse graphics network. In Proc. NIPS2015. 3

Digital Library

[23]

<label>{Lot11}</label> Lottes T.: FXAA. Nvidia White Paper, 2011. 2, 8

[24]

<label>{LSD15}</label> Long J., Shelhamer E., Darrell T.: Fully convolutional networks for semantic segmentation. In Proc. CVPR2015. 3

[25]

<label>{MHBO12}</label> McGuire M., Hennessy P., Bukowski M., Osman B.: A reconstruction filter for plausible motion blur. In Proc. ACM i3D2012, pp. pp.135-42. 2, 8

Digital Library

[26]

<label>{MHN13}</label> Maas A.L., Hannun A.Y., Ng A.Y.: Rectifier nonlinearities improve neural network acoustic models. Proc. ICML Volume 30 2013. 5

[27]

<label>{Mit07}</label> Mittring M.: Finding next gen: Cryengine 2. In ACM SIGGRAPH 2007 Courses2007, pp. pp.97-121. 2

Digital Library

[28]

<label>{NAM*16}</label> Nalbach O., Arabadzhiyska E., Mehta D., Seidel H., Ritschel T.: Deep shading: Convolutional neural networks for screen-space shading. arXiv1603.060782016. 1

[29]

<label>{NKF09}</label> Nowrouzezahrai D., Kalogerakis E., Fiume E.: Shadowing dynamic scenes with arbitrary BRDFs. In Comp. Graph. Forum 2009, vol. Volume 28, pp. pp.249-58. 2

[30]

<label>{NMY15}</label> Narihira T., Maire M., Yu S.X.: Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In Proc. CVPR2015, pp. pp.2992-3. 2

Digital Library

[31]

<label>{OLG*07}</label> Owens J.D., Luebke D., Govindaraju N., Harris M., Krüger J., Lefohn A.E., Purcell T.J.: A survey of general-purpose computation on graphics hardware. In Comp. Graph. Forum 2007, vol. Volume 26, pp. pp.80-113. 2

[32]

<label>{Pho75}</label> Phong B.T.: Illumination for computer generated pictures. Communications of the ACM Volume 18, Issue 6 1975, pp.311-317. 4

Digital Library

[33]

<label>{PVG*11}</label> Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., Blondel M., Prettenhofer P., Weiss R., Dubourg V., VanderPlas J., Passos A., Cournapeau D., Brucher M., Perrot M., Duchesnay E.: Scikit-learn: Machine learning in Python. arXiv1201.04902011. 13

[34]

<label>{RDL*15}</label> Ren P., Dong Y., Lin S., Tong X., Guo B.: Image based relighting using neural networks. ACM Trans. Graph. TOG Volume 34, Issue 4 2015, pp.111. 2, 13

Digital Library

[35]

<label>{RFB15}</label> Ronneberger O., Fischer P., Brox T.: U-Net: Convolutional networks for biomedical image segmentation. In Proc. MICAI. 2015, pp. pp.234-41. 3

[36]

<label>{RGS09}</label> Ritschel T., Grosch T., Seidel H.-P.: Approximating dynamic global illumination in image space. In Proc. ACM i3D2009, pp. pp.75-82. 2, 6, 8, 9

Digital Library

[37]

<label>{Rok93}</label> Rokita P.: Fast generation of depth of field effects in computer graphics. Computers & Graphics Volume 17, Issue 5 1993, pp.593-95. 2

[38]

<label>{RWG*13}</label> Ren P., Wang J., Gong M., Lin S., Tong X., Guo B.: Global illumination with radiance regression functions. ACM Trans. Graph. Proc. SIGGRAPH Volume 32, Issue 4 2013, pp.130. 2, 13

Digital Library

[39]

<label>{ST90}</label> Saito T., Takahashi T.: Comprehensible rendering of 3-d shapes. In ACM SIGGRAPH Computer Graphics 1990, vol. Volume 24, pp. pp.197-206. 2

Digital Library

[40]

<label>{SYM*12}</label> Scherzer D., Yang L., Mattausch O., Nehab D., Sander P.V., Wimmer M., Eisemann E.: Temporal coherence methods in real-time rendering. Comp. Graph. Forum Volume 31, Issue 8 2012, pp.2378-2408. 9

Digital Library

[41]

<label>{TL04}</label> Tabellion E., Lamorlette A.: An approximate global illumination system for computer generated films. ACM Trans. Graph. Proc., SIGGRAPH Volume 23, Issue 3 2004, pp.469-76. 6

Digital Library

[42]

<label>{WFG15}</label> Wang X., Fouhey D.F., Gupta A.: Designing deep networks for surface normal estimation. Proc. CVPR2015. 2

[43]

<label>{Zei12}</label> Zeiler M.D.: ADADELTA: an adaptive learning rate method. CoRR abs/1212.57012012. 5

[44]

<label>{ZGFK17}</label> Zhao H., Gallo O., Frosio I., Kautz J.: Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging Volume 3, Issue 1 2017, pp.47-57. 3, 5

Cited By

Peng JCao ZLuo XXian KTang WZhang JLin G(2025)BokehMe++: Harmonious Fusion of Classical and Neural Rendering for Versatile Bokeh CreationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.350173947:3(1530-1547)Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1109/TPAMI.2024.3501739
Su RDong HRen JJin HChen YWang GLi S(2024)Dynamic Neural Radiosity with Multi-grid DecompositionSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687685(1-12)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687685
Zheng CHuo YHuang HSheng HHuang JTang RZhu HWang RBao H(2024)Neural Global Illumination via Superposed Deformable Feature FieldsSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687680(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687680
Show More Cited By

Deep Shading: Convolutional Neural Networks for Screen Space Shading
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
  2. Computer graphics

Recommendations

Interactive multi-pass programmable shading
SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques

Programmable shading is a common technique for production animation, but interactive programmable shading is not yet widely available. We support interactive programmable shading on virtually any 3D graphics hardware using a scene graph library on top ...
Stochastic Light Culling for VPLs on GGX Microsurfaces

This paper introduces a real-time rendering method for single-bounce glossy caustics created by GGX microsurfaces. Our method is based on stochastic light culling of virtual point lights VPLs, which is an unbiased culling method that randomly determines ...
Two dimensional hidden surface removal with frame-to-frame coherence
SCCG '15: Proceedings of the 31st Spring Conference on Computer Graphics

We describe a hidden surface removal algorithm for two-dimensional layered scenes built from arbitrary primitives, particularly suited to interaction and animation in rich scenes (for example, in illustration). The method makes use of a set-based raster ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computer Graphics Forum

Computer Graphics Forum Volume 36, Issue 4

July 2017

181 pages

ISSN:0167-7055

EISSN:1467-8659

Issue’s Table of Contents

Publisher

The Eurographs Association & John Wiley & Sons, Ltd.

Chichester, United Kingdom

Publication History

Published: 01 July 2017

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Peng JCao ZLuo XXian KTang WZhang JLin G(2025)BokehMe++: Harmonious Fusion of Classical and Neural Rendering for Versatile Bokeh CreationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.350173947:3(1530-1547)Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1109/TPAMI.2024.3501739
Su RDong HRen JJin HChen YWang GLi S(2024)Dynamic Neural Radiosity with Multi-grid DecompositionSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687685(1-12)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687685
Zheng CHuo YHuang HSheng HHuang JTang RZhu HWang RBao H(2024)Neural Global Illumination via Superposed Deformable Feature FieldsSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687680(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687680
Fischer MRitschel T(2024)ZeroGrads: Learning Local Surrogates for Non-Differentiable GraphicsACM Transactions on Graphics10.1145/365817343:4(1-15)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658173
Zeng ZDeschaintre VGeorgiev IHold-Geoffroy YHu YLuan FYan LHašan M(2024)RGB↔X: Image decomposition and synthesis using material- and lighting-aware diffusion modelsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657445(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657445
Fang XZhang S(2024)Single-Video Temporal Consistency Enhancement with Rolling GuidanceComputational Visual Media10.1007/978-981-97-2092-7_6(109-130)Online publication date: 10-Apr-2024
https://dl.acm.org/doi/10.1007/978-981-97-2092-7_6
Valença LZhang JGharbi MHold-Geoffroy YLalonde J(2023)Shadow Harmonization for Realistic CompositingSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618227(1-12)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.1145/3610548.3618227
Zeng CChen GDong YPeers PWu HTong X(2023)Relighting Neural Radiance Fields with Shadow and Highlight HintsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591482(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591482
Gao DMu HXu K(2023)Neural Global Illumination: Interactive Indirect Illumination Prediction Under Dynamic Area LightsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320996329:12(5325-5341)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3209963
Liu CWang LLi ZQuan SXu Y(2023)Real-Time Lighting Estimation for Augmented Reality via Differentiable Screen-Space RenderingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.314194329:4(2132-2145)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3141943
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents