Abstract
Learning and inferring underlying motion patterns of captured 2D scenes and then re-creating dynamic evolution consistent with the real-world natural phenomena have high appeal for graphics and animation. To bridge the technical gap between virtual and real environments, we focus on the inverse modeling and reconstruction of visually consistent and property-verifiable oceans, taking advantage of deep learning and differentiable physics to learn geometry and constitute waves in a self-supervised manner. First, we infer hierarchical geometry using two networks, which are optimized via the differentiable renderer. We extract wave components from the sequence of inferred geometry through a network equipped with a differentiable ocean model. Then, ocean dynamics can be evolved using the reconstructed wave components. Through extensive experiments, we verify that our new method yields satisfactory results for both geometry reconstruction and wave estimation. Moreover, the new framework has the inverse modeling potential to facilitate a host of graphics applications, such as the rapid production of physically accurate scene animation and editing guided by real ocean scenes.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Change history
22 March 2024
An Erratum to this paper has been published: https://doi.org/10.1007/s41095-024-0398-z
References
Nielsen, U. D.; Dietz, J. Ocean wave spectrum estimation using measured vessel motions from an in-service container ship. Marine Structures Vol. 69, 102682, 2020.
Vasavi, S.; Divya, C.; Sarma, A. S. Detection of solitary ocean internal waves from SAR images by using U-Net and KDV solver technique. Global Transitions Proceedings Vol. 2, No. 2, 145–151, 2021.
Pierson, W. J. Jr., Moskowitz, L. A proposed spectral form for fully developed wind seas based on the similarity theory of S. A. Kitaigorodskii. Journal of Geophysical Research Vol. 69, No. 24, 5181–5190, 1964.
Hasselmann, K.; Barnett, T.; Bouws, E.; Carlson, H.; Cartwright, D.; Enke, K.; Ewing, J.; Gienapp, H.; Hasselmann, D.; Kruseman, P.; et al. Measurements of wind-wave growth and swell decay during the Joint North Sea Wave Project (JONSWAP). Report. Deutches Hydrographisches Institut, 1973.
Layton, A. T.; van de Panne, M. A numerically efficient and stable algorithm for animating water waves. The Visual Computer Vol. 18, No. 1, 41–53, 2002.
Max, N. L. Vectorized procedural models for natural terrain. ACM SIGGRAPH Computer Graphics Vol. 15, No. 3, 317–324, 1981.
Peachey, D. R. Modeling waves and surf. ACM SIGGRAPH Computer Graphics Vol. 20, No. 4, 65–74, 1986.
Fournier, A.; Reeves, W. T. A simple model of ocean waves. In: Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, 75–84, 1986.
Hasselmann, D. E.; Dunckel, M.; Ewing, J. A. Directional wave spectra observed during JONSWAP 1973. Journal of Physical Oceanography Vol. 10, No. 8, 1264–1280, 1980.
Bouws, E.; Gunther, H.; Rosenthal, W.; Vincent, C. L. Similarity of the wind wave spectrum in finite depth water: 1. Spectral form. Journal of Geophysical Research: Oceans Vol. 90, No. C1, 975–986, 1985.
Bruneton, E.; Neyret, F.; Holzschuch, N. Real-time realistic ocean lighting using seamless transitions from geometry to BRDF. Computer Graphics Forum Vol. 29, No. 2, 487–496, 2010.
Podee, N.; Max, N.; Iwasaki, K.; Dobashi, Y. Temporal and spatial anti-aliasing for rendering reflections on water waves. Computational Visual Media Vol. 7, No. 2, 201–215, 2021.
Hopper, R.; Wolter, K. The water effects of Pirates of the Caribbean: Dead Men Tell no Tales. In: Proceedings of the ACM SIGGRAPH 2017 Talks, 1–2, 2017.
Huang, L. B.; Qu, Z. Y.; Tan, X.; Zhang, X. X.; Michels, D. L.; Jiang, C. Ships, splashes, and waves on a vast ocean. ACM Transactions on Graphics Vol. 40, No. 6, Article No. 203, 2021.
Xiong, S. Y.; Wang, Z. C.; Wang, M. D.; Zhu, B. A Clebsch method for free-surface vortical flow simulation. ACM Transactions on Graphics Vol. 41, No. 4, Article No. 116, 2022.
Tessendorf, J. Simulating ocean water. In: Simulating Nature: Realistic and Interactive Techniques Course Notes on SIGGRAPH, 3:1–3:26, 2001.
Ashikhmin, M.; Premoze, S.; Shirley, P.; Smits, B. A variance analysis of the Metropolis Light Transport algorithm. Computers & Graphics Vol. 25, No. 2, 287–294, 2001.
Premoze, S.; Ashikhmin, M. Rendering natural waters. Computer Graphics Forum Vol. 20, No. 4, 189–200, 2001.
Hu, Y. H.; Velho, L.; Tong, X.; Guo, B. N.; Shum, H. Realistic, real-time rendering of ocean waves. Computer Animation and Virtual Worlds Vol. 17, No. 1, 59–67, 2006.
Schneider, J.; Westermann, R. Towards real-time visual simulation of water surfaces. In: Proceedings of the Vision Modeling and Visualization Conference, 211–218, 2001.
Loper, M. M.; Black, M. J. OpenDR: An approximate differentiable renderer. In: Computer Vision - ECCV 2014- Lecture Notes in Computer Science, Vol. 8695. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 154–169, 2014.
Henderson, P.; Ferrari, V. Learning single-image 3D reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision Vol. 128, No. 4, 835–854, 2020.
Kato, H.; Ushiku, Y.; Harada, T. Neural 3D mesh renderer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3907–3916, 2018.
Liu, S. C.; Chen, W. K.; Li, T. Y.; Li, H. Soft rasterizer: A differentiable renderer for image-based 3D reasoning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7707–7716, 2019.
Chen, W. Z.; Gao, J.; Ling, H.; Smith, E. J.; Lehtinen, J.; Jacobson, A.; Fidler, S. Learning to predict 3D objects with an interpolation-based differentiable renderer. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Article No. 862, 9609–9619, 2019.
Cole, F.; Genova, K.; Sud, A.; Vlasic, D.; Zhang, Z. T. Differentiable surface rendering via non-differentiable sampling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 6068–6077, 2021.
Li, T. M.; Aittala, M.; Durand, F.; Lehtinen, J. Differentiable Monte Carlo ray tracing through edge sampling. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 222, 2018.
Nimier-David, M.; Vicini, D.; Zeltner, T.; Jakob, W. Mitsuba 2: A retargetable forward and inverse renderer. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 203, 2019.
Girdhar, R.; Fouhey, D. F.; Rodriguez, M.; Gupta, A. Learning a predictable and generative vector representation for objects. In: Computer Vision - ECCV 2016. Lecture Notes in Computer Science, Vol. 9910. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 484–499, 2016.
Hane, C.; Tulsiani, S.; Malik, J. Hierarchical surface prediction for 3D object reconstruction. In: Proceedings of the International Conference on 3D Vision, 412–420, 2017.
Xie, X. G.; Zhai, X.; Hou, F.; Hao, A. M.; Qin, H. Multitask learning on monocular water images: Surface reconstruction and image synthesis. Computer-Animation and Virtual Worlds Vol. 30, Nos. 3–4, e1896, 2019.
Tulsiani, S.; Zhou, T.; Efros, A. A.; Malik, J. Multiview supervision for single-view reconstruction via differentiable ray consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 44, No. 12, 8754–8765, 2022.
Pavlakos, G.; Zhu, L. Y.; Zhou, X. W.; Daniilidis, K. Learning to estimate 3D human pose and shape from a single color image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 459–468, 2018.
Baek, S.; Kim, K. I.; Kim, T. K. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1067–1076, 2019.
Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; Ng, R. NeRF: Representing scenes as neural radiance fields for view synthesis. In: Computer Vision - ECCV 2020. Lecture Notes in Computer Science, Vol. 12346. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 405–421, 2020.
Thapa, S.; Li, N. Y.; Ye, J. W. Dynamic fluid surface reconstruction using deep neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 21–30, 2020.
Li, C.; Qiu, S.; Wang, C. B.; Qin, H. Learning physical parameters and detail enhancement for gaseous scene design based on data guidance. IEEE Transactions on Visualization and Computer Graphics Vol. 27, No. 10, 3867–3880, 2021.
Qiu, S.; Li, C.; Wang, C. B.; Qin, H. A rapid, end-to-end, generative model for gaseous phenomena from limited views. Computer Graphics Forum Vol. 40, No. 6, 242–257, 2021.
He, K. M.; Zhang, X. Y.; Ren, S. Q.; Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778, 2016.
Xie, Y.; Franz, E.; Chu, M. Y.; Thuerey, N. tempoGAN: A temporally coherent, volumetric GAN for superresolution fluid flow. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 95, 2018.
Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234–241, 2015.
Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, Vol. 2, 2672–2680, 2014.
Acknowledgements
This work was sponsored by grants from the National Natural Science Foundation of China (62002010, 61872347), the CAMS Innovation Fund for Medical Sciences (2019-I2M5-016), and the Special Plan for the Development of Distinguished Young Scientists of ISCAS (Y8RC535018).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Xueguang Xie received her B.S. degree in information and computing science from Inner Mongolia University in 2014 and her M.S. degree in software engineering from Beihang University in 2017. She is currently a Ph.D. candidate at the State Key Laboratory of Virtual Reality Technology and Systems, Beihang University. Her research interests include fluid simulation, fluid inverse modeling, and fluid reconstruction.
Yang Gao received his Ph.D. degree in computer science from Beihang University in 2019. He is currently an assistant professor with the State Key Laboratory of Virtual Reality Technology and Systems, Beihang University. His research interests include computer graphics, VR and AR applications, and physics-based modeling and simulation, with a focus on fluid simulation.
Fei Hou received his Ph.D. degree in computer science from Beihang University in 2012. He is currently a research associate professor of the Institute of Software, Chinese Academy of Sciences. He was a postdoctoral researcher at Beihang University from 2012 to 2014, and a research fellow in the School of Computer Science and Engineering, Nanyang Technological University from 2014 to 2017. His research interests include geometry processing, image-based modeling, data vectorization, medical image processing, etc.
Aimin Hao received his B.S., M.S., and Ph.D. degrees in computer science from Beihang University. He is a professor in the School of Computer Science and Engineering, Beihang University, and associate director of the State Key Laboratory of Virtual Reality Technology and Systems. His research interests include virtual reality, computer simulation, computer graphics, geometric modeling, image processing, and computer vision.
Hong Qin received his B.S. and M.S. degrees in computer science from Peking University, and his Ph.D. degree in computer science from the University of Toronto. He is a professor in the Department of Computer Science at Stony Brook University. His research interests include geometric and solid modeling, graphics, physics-based modeling and simulation, computer-aided geometric design, human-computer interaction, visualization, and scientific computing. Currently, he serves as an associate editor for The Visual Computer, Graphical Models, and The Journal of Computer Science and Technology.
Electronic supplementary material
Supplementary material, approximately 85.0 MB.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Xie, X., Gao, Y., Hou, F. et al. Dynamic ocean inverse modeling based on differentiable rendering. Comp. Visual Media 10, 279–294 (2024). https://doi.org/10.1007/s41095-023-0338-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-023-0338-4