[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

Toward wave-based sound synthesis for computer animation

Published: 30 July 2018 Publication History

Abstract

We explore an integrated approach to sound generation that supports a wide variety of physics-based simulation models and computer-animated phenomena. Targeting high-quality offline sound synthesis, we seek to resolve animation-driven sound radiation with near-field scattering and diffraction effects. The core of our approach is a sharp-interface finite-difference time-domain (FDTD) wavesolver, with a series of supporting algorithms to handle rapidly deforming and vibrating embedded interfaces arising in physics-based animation sound. Once the solver rasterizes these interfaces, it must evaluate acceleration boundary conditions (BCs) that involve model-and phenomena-specific computations. We introduce acoustic shaders as a mechanism to abstract away these complexities, and describe a variety of implementations for computer animation: near-rigid objects with ringing and acceleration noise, deformable (finite element) models such as thin shells, bubble-based water, and virtual characters. Since time-domain wave synthesis is expensive, we only simulate pressure waves in a small region about each sound source, then estimate a far-field pressure signal. To further improve scalability beyond multi-threading, we propose a fully time-parallel sound synthesis method that is demonstrated on commodity cloud computing resources. In addition to presenting results for multiple animation phenomena (water, rigid, shells, kinematic deformers, etc.) we also propose 3D automatic dialogue replacement (3DADR) for virtual characters so that pre-recorded dialogue can include character movement, and near-field shadowing and scattering sound effects.

Supplementary Material

ZIP File (109-258.zip)
Supplemental files.
MP4 File (109-258.mp4)
MP4 File (a109-wang.mp4)

References

[1]
T. Akenine-Möller. 2002. Fast 3D Triangle-box Overlap Testing. J. Graph. Tools 6, 1 (2002).
[2]
A. Allen and N. Raghuvanshi. 2015. Aerophones in Flatland: Interactive Wave Simulation of Wind Instruments. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2015) 34, 4 (Aug. 2015).
[3]
S. S. An, D. L. James, and S. Marschner. 2012. Motion-driven Concatenative Synthesis of Cloth Sounds. ACM Transactions on Graphics (SIGGRAPH 2012) (Aug. 2012).
[4]
Avid Technology. 2018. Pro Tools. (2018). http://www.avid.com/pro-tools.
[5]
D. R. Begault. 1994. 3-D Sound for Virtual Reality and Multimedia. Academic Press Professional, Cambridge, MA.
[6]
S. Bilbao. 2009. Numerical Sound Synthesis: Finite Difference Schemes and Simulation in Musical Acoustics. John Wiley and Sons.
[7]
S. Bilbao. 2011. Time domain simulation and sound synthesis for the snare drum. J. Acoust. Soc. Am. 131, 1 (2011).
[8]
Stefan Bilbao. 2013. Modeling of complex geometries and boundary conditions in finite different/finite volume time domain room acoustics simulation. IEEE Transactions on Audio, Speech, and Language Processing 21 (2013).
[9]
S. Bilbao and C. J. Webb. 2013. Physical modeling of timpani drums in 3D on GPGPUs. Journal of the Audio Engineering Society 61, 10 (2013), 737--748.
[10]
N. Bonneel, G. Drettakis, N. Tsingos, I. Viaud-Delmon, and D. James. 2008. Fast Modal Sounds with Scalable Frequency-Domain Synthesis. ACM Transactions on Graphics 27, 3 (Aug. 2008), 24:1--24:9.
[11]
D. Botteldooren. 1994. Acoustical finite-difference time-domain simulation in a quasi-cartesian grid. Journal of the Acoustical Society of America 95 (1994).
[12]
D. Botteldooren. 1997. Time-domain simulation of the influence of close barriers on sound propagation to the environment. The Journal of the Acoustical Society of America 101, 3 (1997), 1278--1285.
[13]
J. N. Chadwick, S. S. An, and D. L. James. 2009. Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells. ACM Transactions on Graphics (Aug. 2009).
[14]
J. N. Chadwick and D. L. James. 2011. Animating Fire with Sound. ACM Transactions on Graphics 30, 4 (Aug. 2011).
[15]
J. N. Chadwick, C. Zheng, and D. L. James. 2012a. Faster Acceleration Noise for Multi-body Animations using Precomputed Soundbanks. ACM Eurographics Symposium on Computer Animation (2012).
[16]
J. N. Chadwick, C. Zheng, and D. L. James. 2012b. Precomputed Acceleration Noise for Improved Rigid-Body Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2012) 31, 4 (Aug. 2012).
[17]
A. Chaigne, C. Touzé, and O. Thomas. 2005. Nonlinear vibrations and chaos in gongs and cymbals. Acoustical science and technology 26, 5 (2005), 403--409.
[18]
A. Chandak, C. Lauterbach, M. Taylor, Z. Ren, and D. Manocha. 2008. Ad-frustum: Adaptive frustum tracing for interactive sound propagation. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1707--1722.
[19]
G. Cirio, D. Li, E. Grinspun, Mi. A. Otaduy, and C. Zheng. 2016. Crumpling sound synthesis. ACM Transactions on Graphics (TOG) 35, 6 (2016), 181.
[20]
R. Clayton and B. Engquist. 1977. Absorbing boundary conditions for acoustic and elastic wave equations. Bulletin of the Seismological Society of America 67, 6 (1977), 1529.
[21]
M. Cook. 2015. Pixar, 'The Road to Point Reyes' and the long history of landscape in new visual technologies. (2015).
[22]
P. R. Cook. 2002. Sound Production and Modeling. IEEE Computer Graphics & Applications 22, 4 (July/Aug. 2002), 23--27.
[23]
R. L. Cook, L. Carpenter, and E. Catmull. 1987. The Reyes Image Rendering Architecture. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '87). ACM, New York, NY, USA, 95--102.
[24]
M. Ducceschi and C. Touzé. 2015. Modal approach for nonlinear vibrations of damped impacted plates: Application to sound synthesis of gongs and cymbals. Journal of Sound and Vibration 344 (2015), 313--331.
[25]
B. Engquist and A. Majda. 1977. Absorbing boundary conditions for numerical simulation of waves. Proceedings of the National Academy of Sciences 74, 5 (1977), 1765--1766.
[26]
Ronald P Fedkiw, Tariq Aslam, Barry Merriman, and Stanley Osher. 1999. A Non-oscillatory Eulerian Approach to Interfaces in Multimaterial Flows (the Ghost Fluid Method). J. Comput. Phys. 152, 2 (1999), 457 -- 492.
[27]
T. Funkhouser, I. Carlbom, G. Elko, G. Pingali, M. Sondhi, and J. West. 1998. A Beam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments. In Proceedings of SIGGRAPH 98 (Computer Graphics Proceedings, Annual Conference Series). 21--32.
[28]
T. A. Funkhouser, P. Min, and I. Carlbom. 1999. Real-Time Acoustic Modeling for Distributed Virtual Environments. In Proceedings of SIGGRAPH 99 (Computer Graphics Proceedings, Annual Conference Series). 365--374.
[29]
W. W. Gaver. 1993. Synthesizing auditory icons. In Proceedings of the TNTERACT'93 and CHI'93 conference on Human factors in computing systems. ACM, 228--235.
[30]
Y. I. Gingold, A. Secord, J. Y. Han, E. Grinspun, and D. Zorin. 2004. A Discrete Model for Inelastic Deformation of Thin Shells.
[31]
G. Guennebaud, B.Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org. (2010).
[32]
Jon Häggblad and Björn Engquist. 2012. Consistent modeling of boundaries in acoustic finite-difference Time-domain simulations. Journal of the Acoustical Society of America 132 (2012).
[33]
P. S. Heckbert. 1987. Ray tracing Jell-O brand gelatin. In ACM SIGGRAPH Computer Graphics, Vol. 21. ACM, 73--74.
[34]
A. Jacobson, D. Panozzo, et al. 2017. libigl: A simple C++ geometry processing library. (2017). http://libigl.github.io/libigl/.
[35]
D. L. James, J. Barbie, and D. K. Pai. 2006. Precomputed Acoustic Transfer: Output-sensitive, accurate sound generation for geometrically complex vibration sources. ACM Transactions on Graphics 25, 3 (July 2006), 987--995.
[36]
D. L. James and D. K. Pai. 2002. DyRT: Dynamic Response Textures for Real Time Deformation Simulation with Graphics Hardware. ACM Trans. Graph. 21, 3 (July 2002), 582--585.
[37]
M. Kleiner, B.-I. Dalenbäck, and P. Svensson. 1993. Auralization-An Overview. J. Audio Engineering Society 41 (1993), 861--861. Issue 11.
[38]
D. Komatitsch, G. Erlebacher, D. Göddeke, and D. Michéa. 2010. High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster. Journal of computational physics 229, 20 (2010), 7692--7714.
[39]
T. R. Langlois, S. S. An, K. K. Jin, and D. L. James. 2014. Eigenmode Compression for Modal Sound Models. ACM Trans. Graph. 33, 4, Article 40 (July 2014), 9 pages.
[40]
T. R Langlois and D. L James. 2014. Inverse-foley animation: Synchronizing rigid-body motions to sound. ACM Transactions on Graphics (TOG) 33, 4 (2014), 41.
[41]
T. R. Langlois, C. Zheng, and D. L. James. 2016. Toward Animating Water with Complex Acoustic Bubbles. ACM Trans. Graph. 35, 4, Article 95 (July 2016), 13 pages.
[42]
S. Larsson and V Thomée. 2009. Partial Differential Equations with Numerical Methods. Springer.
[43]
Q.-H. Liu and J. Tao. 1997. The perfectly matched layer for acoustic waves in absorptive media. The Journal of the Acoustical Society of America 102, 4 (1997), 2072--2082.
[44]
S. Marburg and B. Nolte. 2008. Computational acoustics of noise propagation in fluids: finite and boundary element methods. Vol. 578. Springer.
[45]
R. Mehra, N. Raghuvanshi, L. Antani, A. Chandak, S. Curtis, and D. Manocha. 2013. Wave-based sound propagation in large open scenes using an equivalent source formulation. ACM Transactions on Graphics (TOG) 32, 2 (2013), 19.
[46]
R. Mehra, N. Raghuvanshi, L. Savioja, M. C. Lin, and D. Manocha. 2012. An efficient GPU-based time domain solver for the acoustic wave equation. Applied Acoustics 73, 2(2012), 83--94.
[47]
A. Meshram, R. Mehra, H. Yang, E. Dunn, J.-M. Frahm, and D. Manochak. 2014. P-hrtf: Efficient personalized hrtf computation for high-fidelity spatial sound. Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (2014).
[48]
P. Micikevicius. 2009. 3D Finite Difference Computation on GPUs Using CUDA. In Proceedings of 2Nd Workshop on General Purpose Processing on Graphics Processing Units (GPGPU-2). ACM, New York, NY, USA, 79--84.
[49]
M. Minnaert. 1933. XVI. On musical air-bubbles and the sounds of running water. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 16, 104 (1933), 235--248.
[50]
R. Mittal, H. Dong, M. Bozkurttas, F. M. Najjar, A. Vargas, and A. Loebbecke. 2008. A versatile sharp interface immersed boundary method for incompressible flows with complex boundaries. J. Comput. Phys. 227 (2008).
[51]
R. Mittal and G. Iaccarino. 2005. Immersed Boundary Methods. Annual Review of Fluid Mechanics 37 (2005).
[52]
P. Morse and K. U. Ingard. 1968. Theoretical Acoustics. Princeton University Press, Princeton, New Jersey.
[53]
W. Moss, H. Yeh, J.-M. Hong, M. C. Lin, and D. Manocha. 2010. Sounding Liquids: Automatic Sound Synthesis from Fluid Simulation. ACM Trans. Graph. 29, 3 (2010).
[54]
J. F. O'Brien, P. R. Cook, and G. Essl. 2001. Synthesizing Sounds From Physically Based Motion. In Proceedings of SIGGRAPH 2001. 529--536.
[55]
J. F. O'Brien, C. Shen, and C. M. Gatchalian. 2002. Synthesizing sounds from rigid-body simulations. In The ACM SIGGRAPH 2002 Symposium on Computer Animation. ACM Press, 175--181.
[56]
C. S. Peskin. 1981. The fluid dynamics of heart valves: experimental, theoretical and computational methods. Annual Review of Fluid Mechanics 14 (1981).
[57]
N. Raghuvanshi, R. Narain, and M. C. Lin. 2009. Efficient and Accurate Sound Propagation Using Adaptive Rectangular Decomposition. IEEE Trans. Vis. Comput. Graph. 15, 5 (2009), 789--801.
[58]
N. Raghuvanshi and J. Snyder. 2014. Parametric wave field coding for precomputed sound propagation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 38.
[59]
J. Saarelma, J. Botts, B. Hamilton, and L. Savioja. 2016. Audibility of dispersion error in room acoustic finite-difference time-domain simulation as a function of simulation distance. The Journal of the Acoustical Society of America 139, 4 (2016), 1822--1832.
[60]
C. Schissler, R. Mehra, and D. Manocha. 2014. High-order diffraction and diffuse reflections for interactive sound propagation in large environments. ACM Transactions on Graphics (TOG) 33, 4 (2014), 39.
[61]
C. Schreck, D. Rohmer, D. James, S. Hahmann, and M.-P. Cani. 2016. Real-time sound synthesis for paper material based on geometric analysis. In Eurographics/ACM SIGGRAPH Symposium on Computer Animation (2016).
[62]
E. Schweickart, D. L. James, and S. Marschner. 2017. Animating Elastic Rods with Sound. ACM Transactions on Graphics 36, 4 (July 2017).
[63]
A. A. Shabana. 2012. Theory of Vibration: An Introduction. Springer Science & Business Media.
[64]
A. A. Shabana. 2013. Dynamics of multibody systems. Cambridge university press.
[65]
Side Effects. 2018. Houdini Engine. (2018). http://www.sidefx.com.
[66]
J. O. Smith. 1992. Physical modeling using digital waveguides. Computer music journal 16, 4 (1992), 74--91.
[67]
A. Taflove and S. C. Hagness. 2005. Computational Electrodynamics: The Finite-Difference Time-Domain Method. Artech House.
[68]
T. Takala and J. Hahn. 1992. Sound rendering. In Computer Graphics (Proceedings of SIGGRAPH 92). 211--220.
[69]
J. G. Tolan and J. B. Schneider. 2003. Locally conformal method for acoustic finite-difference time-domain modeling of rigid surfaces. Journal of the Acoustical Society of America 114 (2003).
[70]
N. Tsingos, T. Funkhouser, A. Ngan, and I. Carlbom. 2001. Modeling acoustics in virtual environments using the uniform theory of diffraction. In SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, New York, NY, USA, 545--552.
[71]
K. van den Doel. 2005. Physically Based Models for Liquid Sounds. ACM Trans. Appl. Percept. 2, 4 (Oct. 2005), 534--546.
[72]
K. van den Doel, P. G. Kry, and D. K. Pai. 2001. FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation. (2001), 537--544.
[73]
K. van den Doel and D. K. Pai. 1998. The sounds of physical shapes. Presence: Teleoperators and Virtual Environments 7, 4 (1998), 382--395.
[74]
M. Vorländer. 2008. Auralization. Aachen: Springer (2008).
[75]
C.J. Webb. 2014. Parallel computation techniques for virtual acoustics and physical modelling synthesis. Ph.D. Dissertation.
[76]
C. J. Webb and S. Bilbao. 2011. Computing room acoustics with CUDA - 3D FDTD schemes with boundary losses and viscosity. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2011), 317--320.
[77]
H. Yeh, R. Mehra, Z. Ren, L. Antani, D. Manocha, and M. Lin. 2013. Wave-ray Coupling for Interactive Sound Propagation in Large Complex Scenes. ACM Trans. Graph. 32, 6, Article 165 (Nov. 2013), 11 pages.
[78]
C. Zheng and D. L. James. 2009. Harmonic Fluids. ACM Transactions on Graphics (SIGGRAPH 2009) 28, 3 (Aug. 2009).
[79]
C. Zheng and D. L. James. 2010. Rigid-Body Fracture Sound with Precomputed Sound-banks. ACM Transactions on Graphics (SIGGRAPH 2010) 29, 3 (July 2010).
[80]
C. Zheng and D. L. James. 2011. Toward High-Quality Modal Contact Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2011) 30, 4 (Aug. 2011).

Cited By

View all
  • (2024)WaveBlender: Practical Sound-Source Animation in Blended DomainsSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687696(1-10)Online publication date: 3-Dec-2024
  • (2024)AutoSFX: Automatic Sound Effect Generation for VideosProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681109(9923-9932)Online publication date: 28-Oct-2024
  • (2023)Improved Water Sound Synthesis using Coupled BubblesACM Transactions on Graphics10.1145/359242442:4(1-13)Online publication date: 26-Jul-2023
  • Show More Cited By

Index Terms

  1. Toward wave-based sound synthesis for computer animation

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Graphics
    ACM Transactions on Graphics  Volume 37, Issue 4
    August 2018
    1670 pages
    ISSN:0730-0301
    EISSN:1557-7368
    DOI:10.1145/3197517
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 July 2018
    Published in TOG Volume 37, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. acoustics
    2. computer animation
    3. finite-difference time-domain method
    4. sound synthesis

    Qualifiers

    • Research-article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)37
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 01 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)WaveBlender: Practical Sound-Source Animation in Blended DomainsSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687696(1-10)Online publication date: 3-Dec-2024
    • (2024)AutoSFX: Automatic Sound Effect Generation for VideosProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681109(9923-9932)Online publication date: 28-Oct-2024
    • (2023)Improved Water Sound Synthesis using Coupled BubblesACM Transactions on Graphics10.1145/359242442:4(1-13)Online publication date: 26-Jul-2023
    • (2023)Neural Volumetric Reconstruction for Coherent Synthetic Aperture SonarACM Transactions on Graphics10.1145/359214142:4(1-20)Online publication date: 26-Jul-2023
    • (2023)REALIMPACT: A Dataset of Impact Sound Fields for Real Objects2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00152(1516-1525)Online publication date: Jun-2023
    • (2023)Recycled sand for sustainable 3D-printed sand mold processesThe International Journal of Advanced Manufacturing Technology10.1007/s00170-023-12214-2128:9-10(4049-4060)Online publication date: 25-Aug-2023
    • (2022)Intelligent Animation Creation Method Based on Spatial Separation Perception AlgorithmComputational Intelligence and Neuroscience10.1155/2022/49994782022Online publication date: 1-Jan-2022
    • (2022)GWA: A Large High-Quality Acoustic Dataset for Audio ProcessingACM SIGGRAPH 2022 Conference Proceedings10.1145/3528233.3530731(1-9)Online publication date: 27-Jul-2022
    • (2022)Accurate and Real Time Assisted Cataloging in E-commerce using Dual Images5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)10.1145/3493700.3493738(265-269)Online publication date: 8-Jan-2022
    • (2022)Interactive and Immersive AuralizationSonic Interactions in Virtual Environments10.1007/978-3-031-04021-4_3(77-113)Online publication date: 14-Oct-2022
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media