[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

Event-Based Automatic Differentiation of OpenMP with OpDiLib

Published: 21 March 2023 Publication History

Abstract

We present the new software OpDiLib, a universal add-on for classical operator overloading AD tools that enables the automatic differentiation (AD) of OpenMP parallelized code. With it, we establish support for OpenMP features in a reverse mode operator overloading AD tool to an extent that was previously only reported on in source transformation tools. We achieve this with an event-based implementation ansatz that is unprecedented in AD. Combined with modern OpenMP features around OMPT, we demonstrate how it can be used to achieve differentiation without any additional modifications of the source code; neither do we impose a priori restrictions on the data access patterns, which makes OpDiLib highly applicable. For further performance optimizations, restrictions like atomic updates on adjoint variables can be lifted in a fine-grained manner. OpDiLib can also be applied in a semi-automatic fashion via a macro interface, which supports compilers that do not implement OMPT. We demonstrate the applicability of OpDiLib for a pure operator overloading approach in a hybrid parallel environment. We quantify the cost of atomic updates on adjoint variables and showcase the speedup and scaling that can be achieved with the different configurations of OpDiLib in both the forward and the reverse pass.

References

[1]
D. Auroux and V. Groza. 2017. Optimal parameters identification and sensitivity study for abrasive waterjet milling model. Inverse Probl. Sci. Eng. 25, 11(2017), 1560–1576. DOI:DOI:
[2]
A. Refik Bahadır. 2003. A fully implicit finite-difference scheme for two-dimensional Burgers’ equations. Appl. Math. Comput. 137, 1(2003), 131–137. DOI:DOI:
[3]
Atılım Günes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2017. Automatic differentiation in machine learning: A survey. J. Mach. Learn. Res. 18, 153(2017), 1–43. Retrieved from https://jmlr.org/papers/v18/17-468.html.
[4]
J. Biazar and H. Aminikhah. 2009. Exact and numerical solutions for non-linear Burger’s equation by VIM. Math. Comput. Modell. 49, 7(2009), 1394–1400. DOI:DOI:
[5]
Christian Bischof, Niels Guertler, Andreas Kowarz, and Andrea Walther. 2008. Parallel reverse mode automatic differentiation for OpenMP programs with ADOL-C. In Advances in Automatic Differentiation.Christian H. Bischof, H. Martin Bücker, Paul Hovland, Uwe Naumann, and Jean Utke (Eds.), Springer, Berlin, Germany, 163–173. DOI:DOI:
[6]
H. Martin Bücker, Bruno Lang, Dieter an Mey, and Christian H. Bischof. 2001. Bringing together automatic differentiation and OpenMP. In Proceedings of the 15th International Conference on Supercomputing. ACM, New York, NY, 246–251. DOI:DOI:
[7]
H. Martin Bücker, Bruno Lang, Arno Rasch, Christian H. Bischof, and Dieter an Mey. 2002. Explicit loop scheduling in OpenMP for parallel automatic differentiation. In Proceedings of the 16th Annual International Symposium on High Performance Computing Systems and Applications. IEEE, New York, NY, 121–126. DOI:DOI:
[8]
H. Martin Bücker, Arno Rasch, and Andreas Wolf. 2004. A class of OpenMP applications involving nested parallelism. In Proceedings of the 2004 ACM Symposium on Applied Computing. ACM, New York, NY, 220–224. DOI:DOI:
[9]
Alexandre E. Eichenberger, John Mellor-Crummey, Martin Schulz, Michael Wong, Nawal Copty, Robert Dietrich, Xu Liu, Eugene Loh, and Daniel Lorenz. 2013. OMPT: An OpenMP tools application programming interface for performance analysis. In OpenMP in the Era of Low Power Devices and Accelerators.Alistair P. Rendell, Barbara M. Chapman, and Matthias S. Müller (Eds.), Springer, Berlin, Germany, 171–185. DOI:DOI:
[10]
Michael Förster. 2014. Algorithmic Differentiation of Pragma-Defined Parallel Regions. Springer Vieweg, Wiesbaden, Germany. DOI:DOI:Ph. D. Dissertation. RWTH Aachen University, Aachen, Germany.
[11]
Ralf Giering, Thomas Kaminski, Ricardo Todling, Ronald Errico, Ronald Gelaro, and Nathan Winslow. 2006. Tangent linear and adjoint versions of NASA/GMAO’s Fortran 90 global weather forecast model. In Automatic Differentiation: Applications, Theory, and Implementations.Martin Bücker, George Corliss, Uwe Naumann, Paul Hovland, and Boyana Norris (Eds.), Springer, Berlin, Germany, 275–284. DOI:DOI:
[12]
Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. Vol. 105. SIAM, Philadelphia, PA. DOI:DOI:
[13]
Stefanie Günther, Lars Ruthotto, Jacob B. Schroder, Eric C. Cyr, and Nicolas R. Gauger. 2020. Layer-parallel training of deep residual neural networks. SIAM J. Math. Data Sci. 2, 1(2020), 1–23. DOI:DOI:
[14]
Laurent Hascoët and Valérie Pascual. 2013. The Tapenade automatic differentiation tool: Principles, model, and specification. ACM Trans. Math. Softw. 39, 3(2013), 43 pages. DOI:DOI:
[15]
Patrick Heimbach, Chris Hill, and Ralf Giering. 2002. Automatic generation of efficient adjoint code for a parallel Navier-Stokes solver. In Computational Science.Peter M. A. Sloot, Alfons G. Hoekstra, C. J. Kenneth Tan, and Jack J. Dongarra (Eds.), Springer, Berlin, Germany, 1019–1028. DOI:DOI:
[16]
Patrick Heimbach, Chris Hill, and Ralf Giering. 2005. An efficient exact adjoint of the parallel MIT General Circulation Model, generated via automatic differentiation. Future Gener. Comput. Syst. 21, 8(2005), 1356–1371. DOI:DOI:
[17]
Robin J. Hogan. 2014. Fast reverse-mode automatic differentiation using expression templates in C++. ACM Trans. Math. Softw. 40, 4(2014), 16 pages. DOI:DOI:
[18]
Jan Hückelheim and Laurent Hascoët. 2022. Source-to-source automatic differentiation of OpenMP parallel loops. ACM Trans. Math. Softw. 48, 1(2022), 32 pages. DOI:DOI:
[19]
Jan Hückelheim, Paul Hovland, Michelle Mills Strout, and Jens-Dominik Müller. 2019. Reverse-mode algorithmic differentiation of an OpenMP-parallel compressible flow solver. Int. J. High Perform. Comput. Appl. 33, 1(2019), 140–154. DOI:DOI:
[20]
Tim Kaler, Tao B. Schardl, Brian Xie, Charles E. Leiserson, Jie Chen, Aldo Pareja, and Georgios Kollias. 2021. PARAD: A work-efficient parallel algorithm for reverse-mode automatic differentiation. In Proceedings of the Symposium on Algorithmic Principles of Computer Systems. SIAM, Philadelphia, PA, 144–158. DOI:DOI:
[21]
Andreas Kowarz. 2008. Advanced Concepts for Automatic Differentiation Based on Operator Overloading. Ph.D. Dissertation. Techn. Univ. Dresden, Dresden, Germany. Retrieved from https://nbn-resolving.org/urn:nbn:de:bsz:14-ds-1206719130404-22306.
[22]
Anil Nemili, Emre Özkaya, Nicolas R. Gauger, Felix Kramer, and Frank Thiele. 2016. Discrete adjoint based optimal active control of separation on a realistic high-lift configuration. In New Results in Numerical and Experimental Fluid Mechanics X. Andreas Dillmann, Gerd Heller, Ewald Krämer, Claus Wagner, and Christian Breitsamter (Eds.), Springer, Cham, Switzerland, 237–246. DOI:DOI:
[23]
OpenMP Architecture Review Board. 2005. OpenMP Application Program Interface Version 2.5. (May2005). Retrieved from https://www.openmp.org/wp-content/uploads/spec25.pdf.
[24]
OpenMP Architecture Review Board. 2013. OpenMP Application Program Interface Version 4.0. (July2013). Retrieved from https://www.openmp.org/wp-content/uploads/OpenMP4.0.0.pdf.
[25]
OpenMP Architecture Review Board. 2018. OpenMP Application Programming Interface Version 5.0. (November2018). Retrieved from https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5.0.pdf.
[26]
OpenMP Architecture Review Board. 2021. OpenMP Application Programming Interface Version 5.2. (November2021). Retrieved from https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf.
[27]
Emre Özkaya and Nicolas R. Gauger. 2010. Automatic transition from simulation to one-shot shape optimization with Navier-Stokes equations. GAMM-Mitteilungen 33, 2(2010), 133–147. DOI:DOI:
[28]
Joachim Protze, Tim Cramer, Simon Convent, and Matthias S. Müller. 2019. OMPT-Multiplex: Nesting of OMPT tools. In Tools for High Performance Computing 2017.Christoph Niethammer, Michael M. Resch, Wolfgang E. Nagel, Holger Brunst, and Hartmut Mix (Eds.), Springer, Cham, Switzerland, 73–83. DOI:DOI:
[29]
M. Sagebaum, T. Albring, and N. R. Gauger. 2018. Expression templates for primal value taping in the reverse mode of algorithmic differentiation. Optim. Methods Software 33, 4–6(2018), 1207–1231. DOI:DOI:
[30]
Max Sagebaum, Tim Albring, and Nicolas R. Gauger. 2019. High-performance derivative computations using CoDiPack. ACM Trans. Math. Softw. 45, 4(2019), 26 pages. DOI:DOI:
[31]
Max Sagebaum, Johannes Blühdorn, and Nicolas R. Gauger. 2020. Index Handling and Assign Optimization for Algorithmic Differentiation Reuse Index Managers. (2020). arXiv:2006.12992. Retrieved from https://arxiv.org/abs/2006.12992.
[32]
Max Sagebaum, Nicolas R. Gauger, Uwe Naumann, Johannes Lotz, and Klaus Leppkes. 2013. Algorithmic differentiation of a complex C++ code with underlying libraries, In Proceedings of the 2013 International Conference on Computational Science. Vassil Alexandrov, Michael Lees, Valeria Krzhizhanovskaya, Jack Dongarra, and Peter M. A. Sloot (Eds.), 208–217. DOI:
[33]
Michel Schanen, Michael Förster, Johannes Lotz, Klaus Leppkes, and Uwe Naumann. 2012. Adjoining hybrid parallel code. In Proceedings of the 8th International Conference on Engineering Computational Technology.B. H. V. Topping (Ed.), Vol. 100, Civil-Comp Press, Kippen, Stirlingshire, UK, 18. DOI:DOI:
[34]
Michel Schanen, Uwe Naumann, Laurent Hascoët, and Jean Utke. 2010. Interpretative adjoints for numerical simulation codes using MPI, In 2010 International Conference on Computational Science. Procedia Comput. Sci. 1, 1 (2010), 1825–1833. DOI:DOI:
[35]
Sebastian Schlenkrich, Andrea Walther, Nicolas R. Gauger, and Ralf Heinrich. 2008. Differentiating fixed point iterations with ADOL-C: Gradient calculation for fluid dynamics. In Modeling, Simulation and Optimization of Complex Processes. Hans Georg Bock, Ekaterina Kostina, Hoang Xuan Phu, and Rolf Rannacher (Eds.), Springer, Berlin, Germany, 499–508. DOI:DOI:
[36]
Markus Towara and Uwe Naumann. 2013. A discrete adjoint model for OpenFOAM, In Proceedings of the 2013 International Conference on Computational Science. Vassil Alexandrov, Michael Lees, Valeria Krzhizhanovskaya, Jack Dongarra, and Peter M. A. Sloot (Eds.), 429–438. DOI:
[37]
Tom Verstraete, Lasse Müller, and Jens-Dominik Müller. 2017. Adjoint-based design optimisation of an internal cooling channel u-bend for minimised pressure losses. Int. J. Turbomach. Propuls. Power 2, 2(2017), 13. DOI:DOI:
[38]
S. Vitale, M. Pini, and P. Colonna. 2020. Multistage turbomachinery design using the discrete adjoint method within the open-source software SU2. Journal Propul. Power 36, 3(2020), 465–478. DOI:DOI:
[39]
Andrea Walther. 2009. Getting started with ADOL-C. In Combinatorial Scientific Computing.Uwe Naumann, Olaf Schenk, Horst D. Simon, and Sivan Toledo (Eds.), Vol. 9061, Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 10. Retrieved from https://drops.dagstuhl.de/opus/volltexte/2009/2084.
[40]
Hongqing Zhu, Huazhong Shu, and Meiyu Ding. 2010. Numerical solutions of two-dimensional Burgers’ equations by discrete Adomian decomposition method. Comput. Math. Appl. 60, 3(2010), 840–848. DOI:

Cited By

View all

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Mathematical Software
ACM Transactions on Mathematical Software  Volume 49, Issue 1
March 2023
250 pages
ISSN:0098-3500
EISSN:1557-7295
DOI:10.1145/3587918
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 March 2023
Online AM: 17 November 2022
Accepted: 22 October 2022
Revised: 30 June 2022
Received: 28 September 2021
Published in TOMS Volume 49, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Algorithmic differentiation
  2. OpenMP
  3. OMPT
  4. high performance computing
  5. AD tool design
  6. reusable software

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)69
  • Downloads (Last 6 weeks)9
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media