More Web Proxy on the site http://driver.im/

research-article

A next-generation discontinuous galerkin fluid dynamics solver with application to high-resolution lung airflow simulations

Authors:

Martin Kronbichler,

Maximilian Bergbauer,

Karl-Robert Wichmann,

Carolin Geitner,

Wolfgang A. WallAuthors Info & Claims

SC '21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 21, Pages 1 - 15

https://doi.org/10.1145/3458817.3476171

Published: 13 November 2021 Publication History

Abstract

We present a novel, highly scalable and optimized solver for turbulent flows based on high-order discontinuous Galerkin discretizations of the incompressible Navier-Stokes equations aimed to minimize time-to-solution. The solver uses explicit-implicit time integration with variable step size. The central algorithmic component is the matrix-free evaluation of discretized finite element operators. The node-level performance is optimized by sum-factorization kernels for tensor-product elements with unique algorithmic choices that reduce the number of arithmetic operations, improve cache usage, and vectorize the arithmetic work across elements and faces. These ingredients are integrated into a framework scalable to the massive parallelism of supercomputers by the use of optimal-complexity linear solvers, such as mixed-precision, hybrid geometric-polynomial-algebraic multigrid solvers for the pressure Poisson problem. The application problem under consideration are fluid dynamical simulations of the human respiratory system under mechanical ventilation conditions, using unstructured/structured adaptively refined meshes for geometrically complex domains typical of biomedical engineering.

Supplementary Material

MP4 File (A Next-Gen Discontinuous Galerkin Fluid Dynamics Solver w_ Application to High-Resolution Lung Airflow Simulations.mp4)

Presentation video

Download
311.32 MB

References

[1]

A. Abdelfattah, V. Barra, N. Beams, J. Brown, J.-S. Camier, V. Dobrev, Y. Dudouit, L. Ghaffari, T. Kolev, D. Medina, T. Rathnayake, J. L. Thompson, and S. Tomov. 2020. libCEED user manual.

[2]

M. Adams, M. Brezina, J. Hu, and R. Tuminaro. 2003. Parallel multigrid smoothing: polynomial versus Gauss-Seidel. J. Comput. Phys. 188 (2003), 593--610.

Digital Library

[3]

D. Arndt, W. Bangerth, B. Blais, T. C. Clevenger, M. Fehling, A. V. Grayver, T. Heister, L. Heltai, M. Kronbichler, M. Maier, P. Munch, J.-P. Pelteret, R. Rastak, I. Tomas, B. Turcksin010920, Z. Wang, and D. Wells. 2020. The deal.II library, version 9.2. J. Numer. Math. 28, 3 (2020), 131--146.

[4]

D. Arndt, W. Bangerth, B. Blais, M. Fehling, R. Gassmöller, T. Heister, L. Heltai, U. Köcher, M. Kronbichler, M. Maier, P. Munch, J.-P. Pelteret, S. Proell, K. Simon, B. Turcksin, D. Wells, and J. Zhang. 2021. The deal.II library, version 9.3. J. Numer. Math. 29, 3 (2021).

[5]

D. Arndt, W. Bangerth, D. Davydov, T. Heister, L. Heltai, M. Kronbichler, M. Maier, J.-P. Pelteret, B. Turcksin, and D. Wells. 2021. The deal.II finite element library: design, features, and insights. Comput. Math. Appl. 81 (2021), 407--422.

[6]

D. Arndt, N. Fehn, G. Kanschat, K. Kormann, M. Kronbichler, P. Munch, W. A. Wall, and J. Witte. 2020. ExaDG - high-order discontinuous Galerkin for the exascale. In Software for Exascale Computing - SPPEXA 2016--2019 (Lecture Notes in Computational Science and Engineering 136), H.-J. Bungartz, S. Reiz, B. Uekermann, P. Neumann, and W. E. Nagel (Eds.). Springer International Publishing, Cham, 189--224.

[7]

W. Bangerth, C. Burstedde, T. Heister, and M. Kronbichler. 2011. Algorithms and data structures for massively parallel generic finite element codes. ACM Trans. Math. Softw. 38, 2 (2011), 14:1--28.

Digital Library

[8]

J. Bates. 2009. Lung Mechanics: An Inverse Modeling Approach. Cambridge University Press, Cambridge, UK New York.

[9]

R. Becker and M. Braack. 2000. Multigrid techniques for finite elements on locally refined meshes. Numer. Lin. Algebr. Appl. 7, 6 (2000), 363--379. &lang;363::aid-nla202&rang;3.0.co;2-v

[10]

R. G. Brower, M. A. Matthay, A. Morris, D. Schoenfeld, B. T. Thompson, A. Wheeler, H. P. Wiedemann, A. C. Arroliga, C. J. Fisher, J. J. Komara, et al. 2000. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. New Engl. J. Med. 342, 18 (2000), 1301--1308.

[11]

J. Brown. 2010. Efficient nonlinear solvers for nodal high-order finite elements in 3D. J. Sci. Comput. 45, 1--3 (2010), 48--63.

Digital Library

[12]

C. Burstedde, L. C. Wilcox, and O. Ghattas. 2011. p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees. SIAM J. Sci. Comput. 33, 3 (2011), 1103--1133.

Digital Library

[13]

J. Choi, G. Xia, M. H. Tawhai, E. A. Hoffman, and C.-L. Lin. 2010. Numerical study of high-frequency oscillatory air flow and convective mixing in a CT-based human airway model. Ann. Biomed. Eng. 38, 12 (2010), 3550--3571.

[14]

T. C. Clevenger, T. Heister, G. Kanschat, and M. Kronbichler. 2021. A flexible, parallel, adaptive geometric multigrid method for FEM. ACM Trans. Math. Softw. 47, 1 (2021), 7:1--27.

Digital Library

[15]

J. W. De Backer, W. G. Vos, S. C. Vinchurkar, R. Claes, A. Drollmann, D. Wulfrank, P. M. Parizel, P. Germonpré, and W. De Backer. 2010. Validation of computational fluid dynamics in CT-based airway models with SPECT/CT. Radiology 257, 3 (2010), 854--862.

[16]

M. O. Deville, P. F. Fischer, and E. H. Mund. 2002. High-order methods for incompressible fluid flow. Vol. 9. Cambridge University Press.

[17]

A. Esteban, A. Anzueto, F. Frutos, I. Alía, L. Brochard, T. E. Stewart, S. Benito, S. K. Epstein, C. Apezteguía, P. Nightingale, et al. 2002. Characteristics and outcomes in adult patients receiving mechanical ventilation: a 28-day international study. JAMA 287, 3 (2002), 345--355.

[18]

R. D. Falgout, J. E. Jones, and U. M. Yang. [n.d.]. The design and implementation of hypre, a library of parallel high performance preconditioners. In Lecture Notes in Computational Science and Engineering. Springer-Verlag, 267--294.

[19]

N. Fehn, J. Heinz, W. A. Wall, and M. Kronbichler. 2021. High-order arbitrary Lagrangian-Eulerian discontinuous Galerkin methods for the incompressible Navier-Stokes equations. J. Comput. Phys. 430 (2021), 110040.

[20]

N. Fehn, M. Kronbichler, C. Lehrenfeld, G. Lube, and P. W. Schroeder. 2019. High-order DG solvers for under-resolved turbulent incompressible flows: A comparison of L² and H(div) methods. Int. J. Numer. Meth. Fluids 91, 11 (2019), 533--556.

[21]

N. Fehn, M. Kronbichler, P. Munch, and M. Bergbauer. 2021. ExaDG: High-Order Discontinuous Galerkin for the Exa-Scale. Code available on https://github.com/exadg/exadg.

Digital Library

[22]

N. Fehn, P. Munch, W. A. Wall, and M. Kronbichler. 2020. Hybrid multigrid methods for high-order discontinuous Galerkin discretizations. J. Comput. Phys. 415 (2020), 109538.

[23]

N. Fehn, W. A. Wall, and M. Kronbichler. 2017. On the stability of projection methods for the incompressible Navier-Stokes equations based on high-order discontinuous Galerkin discretizations. J. Comput. Phys. 351 (2017), 392--421.

[24]

N. Fehn, W. A. Wall, and M. Kronbichler. 2018. Efficiency of high-performance discontinuous Galerkin spectral element methods for under-resolved turbulent incompressible flows. Int. J. Numer. Meth. Fluids 88, 1 (2018), 32--54.

[25]

N. Fehn, W. A. Wall, and M. Kronbichler. 2018. Robust and efficient discontinuous Galerkin methods for under-resolved turbulent incompressible flows. J. Comput. Phys. 372 (2018), 667--693.

Digital Library

[26]

T. Ferkol and D. Schraufnagel. 2014. The global burden of respiratory disease. Ann. Amer. Thorac. Soc. 11, 3 (2014), 404--406.

[27]

P. Fischer, M. Min, T. Rathnayake, S. Dutta, T. Kolev, V. Dobrev, J.-S. Camier, M. Kronbichler, T. Warburton, K. Świrydowicz, and J. Brown. 2020. Scalability of high-performance PDE solvers. Int. J. High Perf. Comput. Appl. 34, 5 (2020), 562--586.

Digital Library

[28]

P. F. Fischer, S. Kerkemeier, et al. 2020. Nek5000 Web page. https://nek5000.mcs.anl.gov.

[29]

T. Gemci, V. Ponyavin, Y. Chen, H. Chen, and R. Collins. 2008. Computational model of airflow in upper 17 generations of human respiratory tract. J. Biomech. 41, 9 (2008), 2047--2054.

[30]

W. J. Gordon and L. C. Thiel. 1982. Transfinite mappings and their application to grid generation. Appl. Math. Comput. 10 (1982), 171--233.

Digital Library

[31]

J. Guttmann, L. Eberhard, B. Fabry, W. Bertschmann, and G. Wolff. 1993. Continuous calculation of intratracheal pressure in tracheally intubated patients. Anesthesiology 79, 3 (1993), 503--513.

[32]

L. Heltai, W. Bangerth, M. Kronbichler, and A. Mola. 2021. Propagating geometry information to finite element computations. ACM Trans. Math. Softw. 47, 3 (2021), in press.

Digital Library

[33]

J. Hoberock. 2019. Working Draft, C ++ Extensions for Parallelism Version 2. Technical Report.

[34]

K. Horsfield and G. Cumming. 1968. Morphology of the bronchial tree in man. Journal of Applied Physiology 24, 3 (1968), 373--383.

[35]

S. Kabilan, C.-L. Lin, and E. A. Hoffman. 2007. Characteristics of airflow in a CT-based ovine lung: a numerical study. J. Appl. Physiol. 102, 4 (2007), 1469--1482.

[36]

G. E. Karniadakis, M. Israeli, and S. A. Orszag. 1991. High-order splitting methods for the incompressible Navier-Stokes equations. J. Comput. Phys. 97, 2 (1991), 414--443.

[37]

C. Kato, Y. Yamade, K. Nagano, K. Kumahata, K. Minami, and T. Nishikawa. 2020. Toward realization of numerical towing-tank tests by wall-resolved large eddy simulation based on 32 billion grid finite-element computation. In 2020 SC20: International Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE Computer Society, 23--35.

[38]

D. Kempf, R. Heß, S. Müthing, and P. Bastian. 2021. Automatic code generation for high-performance discontinuous Galerkin methods on modern architectures. ACM Trans. Math. Softw. 47, 1 (2021), 6/1--31.

Digital Library

[39]

T. Kolev et al. 2020. Support CEED-enabled ECP applications in their preparation for Aurora/Frontier. Technical Report ECP Milestone CEED-MS35. US Department of Energy.

[40]

T. Kolev et al. 2021. High-order algorithmic developments and optimizations for large-scale GPU-accelerated simulations. Technical Report ECP Milestone CEED-MS36. US Department of Energy.

[41]

B. Krank, N. Fehn, W. A. Wall, and M. Kronbichler. 2017. A high-order semi-explicit discontinuous Galerkin solver for 3D incompressible flow with application to DNS and LES of turbulent channel flow. J. Comput. Phys. 348 (2017), 634--659.

Digital Library

[42]

M. Kronbichler and K. Kormann. 2012. A generic interface for parallel cell-based finite element operator application. Comput. Fluids 63 (2012), 135--147.

[43]

M. Kronbichler and K. Kormann. 2019. Fast matrix-free evaluation of discontinuous Galerkin finite element operators. ACM Trans. Math. Softw. 45, 3 (2019), 29:1--40.

Digital Library

[44]

M. Kronbichler and K. Ljungkvist. 2019. Multigrid for matrix-free high-order finite element computations on graphics processors. ACM Trans. Parallel Comput. 6, 1 (2019), 2:1--32.

Digital Library

[45]

M. Kronbichler and W. A. Wall. 2018. A performance comparison of continuous and discontinuous Galerkin methods with fast multigrid solvers. SIAM J. Sci. Comput. 40, 5 (2018), A3423--A3448.

Digital Library

[46]

K. Ljungkvist. 2017. Matrix-free Finite-element Computations on Graphics Processors with Adaptively Refined Unstructured Meshes. In Proceedings of the 25th High Performance Computing Symposium (Virginia Beach, Virginia) (HPC '17). Society for Computer Simulation International, San Diego, CA, USA, Article 1, 12 pages. http://dl.acm.org/citation.cfm?id=3108096.3108097

Digital Library

[47]

R. Löhner. 2019. Towards overcoming the LES crisis. Int. J. Comput. Fluid Dyn. 33, 3 (2019), 87--97.

[48]

M. G. Ménache, W. Hofmann, B. Ashgarian, and F. J. Miller. 2008. Airway geometry models of children's lungs for use in dosimetry modeling. Inhalation toxicology 20, 2 (2008), 101--126.

[49]

T. Möller and B. Trumbore. 1997. Fast, minimum storage ray-triangle intersection. Journal of Graphics Tools 2, 1 (1997), 21--28. arXiv:https://doi.org/10.1080/10867651.1997.10487468

Digital Library

[50]

D. Moxey, R. Amici, and M. Kirby. 2020. Efficient matrix-free high-order finite element evaluation for simplicial elements. SIAM J. Sci. Comput. 42, 3 (2020), C97--C123.

Digital Library

[51]

N. Offermans, O. Marin, M. Schanen, J. Gong, P. Fischer, P. Schlatter, A. Obabko, A. Peplinski, M. Hutchinson, and E. Merzari. 2016. On the strong scaling of the spectral element solver Nek5000 on petascale systems. In Proceedings of the Exascale Applications and Software Conference 2016 (Stockholm, Sweden) (EASC '16). ACM, New York, NY, USA, Article 5, 10 pages.

Digital Library

[52]

S. A. Orszag. 1980. Spectral methods for problems in complex geometries. J. Comput. Phys. 37 (1980), 70--92.

[53]

H.-C. Pape, A. Kurtz, and S. Silbernagl. 2018. Physiologie. Georg Thieme Verlag, Stuttgart New York.

[54]

S. Qi, B. Zhang, Y. Teng, J. Li, Y. Yue, Y. Kang, and W. Qian. 2017. Transient dynamics simulation of airflow in a CT-scanned human airway tree: more or fewer terminal bronchi? Comput. Math. Method Med. 2017 (2017).

[55]

C. J. Roth, K. M. Förster, A. Hilgendorff, B. Ertl-Wagner, W. A. Wall, and A. W. Flemmer. 2018. Gas exchange mechanisms in preterm infants on HFOV-a computational approach. Scientific reports 8, 1 (2018), 1--8.

[56]

H. Sundar, G. Biros, C. Burstedde, J. Rudi, O. Ghattas, and G. Stadler. 2012. Parallel geometric-algebraic multigrid on unstructured forests of octrees. In SC'12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. IEEE, 1--11.

[57]

M. Tawhai, A. J. Pullan, and P. Hunter. 2000. Generation of an anatomically based three-dimensional model of the conducting airways. Annals of biomedical engineering 28 (08 2000), 793--802.

[58]

J. Treibig, G. Hager, and G. Wellein. 2010. LIKWID: A lightweight performance-oriented tool suite for x86 multicore environments. In Proceedings of PSTI2010, the First International Workshop on Parallel Software Tools and Tool Infrastructures. San Diego CA. https://github.com/RRZE-HPC/likwid.

Digital Library

[59]

D. K. Walters, G. W. Burgreen, D. M. Lavallee, D. S. Thompson, and R. L. Hester. 2011. Efficient, physiologically realistic lung airflow simulations. IEEE T. Biomed. Eng. 58, 10 (2011), 3016--3019.

[60]

E. R. Weibel, A. F. Cournand, and D. W. Richards. 1963. Morphometry of the human lung. Vol. 1. Springer.

[61]

J. B. West and A. M. Luks. 2016. West's Respiratory Physiology: The Essentials. Wolters Kluwer, Philadelphia.

[62]

S. Williams, A. Waterman, and D. Patterson. 2009. Roofline: An insightful visual performance model for multicore architectures. Commun. ACM 52, 4 (2009), 65--76.

Digital Library

Cited By

Proell SMunch PKronbichler MWall WMeier C(2024)A highly efficient computational approach for fast scan-resolved simulations of metal additive manufacturing processes on the scale of real partsAdditive Manufacturing10.1016/j.addma.2023.10392179(103921)Online publication date: Jan-2024
https://doi.org/10.1016/j.addma.2023.103921
Munch PHeister TPrieto Saavedra LKronbichler M(2023)Efficient Distributed Matrix-free Multigrid Methods on Locally Refined Meshes for FEM ComputationsACM Transactions on Parallel Computing10.1145/358031410:1(1-38)Online publication date: 29-Mar-2023
https://dl.acm.org/doi/10.1145/3580314
Davis JShafner JNichols DGrube NMartin PBhatele A(2023)Porting a Computational Fluid Dynamics Code with AMR to Large-scale GPU Platforms2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS54959.2023.00066(602-612)Online publication date: May-2023
https://doi.org/10.1109/IPDPS54959.2023.00066
Show More Cited By

Index Terms

A next-generation discontinuous galerkin fluid dynamics solver with application to high-resolution lung airflow simulations
1. Applied computing
  1. Life and medical sciences
    1. Health informatics
2. Mathematics of computing
  1. Mathematical software
    1. Mathematical software performance
    2. Solvers

Recommendations

On the eddy-resolving capability of high-order discontinuous Galerkin approaches to implicit LES / under-resolved DNS of Euler turbulence

We present estimates of spectral resolution power for under-resolved turbulent Euler flows obtained with high-order discontinuous Galerkin (DG) methods. The '1% rule' based on linear dispersion-diffusion analysis introduced by Moura et al. (2015) 10 is ...
Analysis of Pulmonary Airflow on a Smartphone Application
A high-order semi-explicit discontinuous Galerkin solver for 3D incompressible flow with application to DNS and LES of turbulent channel flow

We present an efficient discontinuous Galerkin scheme for simulation of the incompressible NavierStokes equations including laminar and turbulent flow. We consider a semi-explicit high-order velocity-correction method for time integration as well as ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2021

1493 pages

ISBN:9781450384421

DOI:10.1145/3458817

General Chair:
Bronis R. de Supinski,
Program Chairs:
Mary Hall,
Todd Gamblin

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Artifacts Available / v1.1

Author Tags

Qualifiers

Research-article

Funding Sources

Gauss Centre for Supercomputing e.V.
Deutsche Forschungsgemeinschaft
Bayerische Kompetenznetzwerk für Technisch-Wissenschaftliches Hoch- und Höchstleistungsrechnen (KONWIHR)

Conference

SC '21

Sponsor:

SIGHPC

SC '21: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 14 - 19, 2021

Missouri, St. Louis

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
478
Total Downloads

Downloads (Last 12 months)66
Downloads (Last 6 weeks)10

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Proell SMunch PKronbichler MWall WMeier C(2024)A highly efficient computational approach for fast scan-resolved simulations of metal additive manufacturing processes on the scale of real partsAdditive Manufacturing10.1016/j.addma.2023.10392179(103921)Online publication date: Jan-2024
https://doi.org/10.1016/j.addma.2023.103921
Munch PHeister TPrieto Saavedra LKronbichler M(2023)Efficient Distributed Matrix-free Multigrid Methods on Locally Refined Meshes for FEM ComputationsACM Transactions on Parallel Computing10.1145/358031410:1(1-38)Online publication date: 29-Mar-2023
https://dl.acm.org/doi/10.1145/3580314
Davis JShafner JNichols DGrube NMartin PBhatele A(2023)Porting a Computational Fluid Dynamics Code with AMR to Large-scale GPU Platforms2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS54959.2023.00066(602-612)Online publication date: May-2023
https://doi.org/10.1109/IPDPS54959.2023.00066
Arndt DBangerth WFeder MFehling MGassmöller RHeister THeltai LKronbichler MMaier MMunch PPelteret JSticko STurcksin BWells D(2022)The deal.II library, Version 9.4Journal of Numerical Mathematics10.1515/jnma-2022-005430:3(231-246)Online publication date: 17-Jul-2022
https://doi.org/10.1515/jnma-2022-0054
Dai ZDeng LWang YWang FLi MZhang J(2022)Performance Optimization and Analysis of the Unstructured Discontinuous Galerkin Solver on Multi-Core and Many-Core Architectures2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys)10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00158(993-999)Online publication date: Dec-2022
https://doi.org/10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00158
Munch PLjungkvist KKronbichler M(2022)Efficient Application of Hanging-Node Constraints for Matrix-Free High-Order FEM Computations on CPU and GPUHigh Performance Computing10.1007/978-3-031-07312-0_7(133-152)Online publication date: 29-May-2022
https://dl.acm.org/doi/10.1007/978-3-031-07312-0_7

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents