More Web Proxy on the site http://driver.im/

research-article

Enabling and scaling biomolecular simulations of 100 million atoms on petascale machines with a multicore-optimized message-driven runtime

Authors:

Laxmikant V. Kale,

James C. Phillips,

Chris HarrisonAuthors Info & Claims

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 61, Pages 1 - 11

https://doi.org/10.1145/2063384.2063466

Published: 12 November 2011 Publication History

Abstract

A 100-million-atom biomolecular simulation with NAMD is one of the three benchmarks for the NSF-funded sustainable petascale machine. Simulating this large molecular system on a petascale machine presents great challenges, including handling I/O, large memory footprint and getting good strong-scaling results. In this paper, we present parallel I/O techniques to enable the simulation. A new SMP model is designed to efficiently utilize ubiquitous wide multicore clusters by extending the Charm++ asynchronous message-driven runtime. We exploit node-aware techniques to optimize both the application and the underlying SMP runtime. Hierarchical load balancing is further exploited to scale NAMD to the full Jaguar PF Cray XT5 (224,076 cores) at Oak Ridge National Laboratory, both with and without PME full electrostatics, achieving 93% parallel efficiency (vs 6720 cores) at 9 ms per step for a simple cutoff calculation. Excellent scaling is also obtained on 65,536 cores of the Intrepid Blue Gene/P at Argonne National Laboratory.

References

[1]

S. R. Alam, R. F. Barrett, J. A. Kuehn, P. C. Roth, and J. S. Vetter. Characterization of scientific workloads on systems with multi-core processors. In In IISWC, pages 225--236, 2006.

[2]

A. Bhatele, S. Kumar, C. Mei, J. C. Phillips, G. Zheng, and L. V. Kale. Overcoming scaling challenges in biomolecular simulations across multiple platforms. In Proceedings of IEEE International Parallel and Distributed Processing Symposium 2008, pages 1--12, April 2008.

Digital Library

[3]

A. Bhatele, L. Wesolowski, E. Bohm, E. Solomonik, and L. V. Kale. Understanding application performance via micro-benchmarks on three large supercomputers: Intrepid, Ranger and Jaguar. International Journal of High Performance Computing Applications (IHHPCA), 24(4):411--427, 2010.

Digital Library

[4]

E. Bohm, A. Bhatele, L. V. Kale, M. E. Tuckerman, S. Kumar, J. A. Gunnels, and G. J. Martyna. Fine Grained Parallelization of the Car-Parrinello ab initio MD Method on Blue Gene/L. IBM Journal of Research and Development: Applications of Massively Parallel Systems, 52(1/2):159--174, 2008.

Digital Library

[5]

K. J. Bowers, E. Chow, H. Xu, R. O. Dror, M. P. Eastwood, B. A. Gregersen, J. L. Klepeis, I. Kolossvary, M. A. Moraes, F. D. Sacerdoti, J. K. Salmon, Y. Shan, and D. E. Shaw. Molecular dynamics---scalable algorithms for molecular dynamics simulations on commodity clusters. In SC '06: Proceedings of the 2006 ACM/IEEE conference on Supercomputing, page 84, New York, NY, USA, 2006. ACM Press.

Digital Library

[6]

S. Browne, J. Dongarra, N. Garner, G. Ho, and P. Mucci. A Portable Programming Interface for Performance Evaluation on Modern Processors. Int. J. High Perform. Comput. Appl., 14(3):189--204, 2000.

Digital Library

[7]

D. A. Case, T. E. Cheatham, T. Darden, H. Gohlke, R. Luo, K. M. Merz, A. Onufriev, C. Simmerling, B. Wang, and R. J. Woods. The Amber biomolecular simulation programs. J. Comput. Chem., 26(16):1668--1688, Dec. 2005.

[8]

B. G. Fitch, A. Rayshubskiy, M. Eleftheriou, T. J. C. Ward, M. Giampapa, and M. C. Pitman. Blue matter: Approaching the limits of concurrency for classical molecular dynamics. In SC '06: Proceedings of the 2006 ACM/IEEE conference on Supercomputing, New York, NY, USA, 2006. ACM Press.

Digital Library

[9]

B. Hess, C. Kutzner, D. van der Spoel, and E. Lindahl. Gromacs 4: Algorithms for highly efficient, load-balanced, and scalable molecular simulation. Journal of Chemical Theory and Computation, 4(3):435--447, 2008.

[10]

W. F. Humphrey, A. Dalke, and K. Schulten. VMD -- Visual molecular dynamics. Journal of Molecular Graphics, 14(1):33--38, 1996.

[11]

P. Jetley, F. Gioachin, C. Mendes, L. V. Kale, and T. R. Quinn. Massively parallel cosmological simulations with ChaNGa. In Proceedings of IEEE International Parallel and Distributed Processing Symposium 2008, pages 1--12, 2008.

[12]

M. Jones, R. Yao, and C. Bhole. Hybrid mpi-openmp programming for parallel osem pet reconstruction. Nuclear Science, IEEE Transactions on, 53(5):2752--2758, oct. 2006.

[13]

L. V. Kale and G. Zheng. Charm++ and AMPI: Adaptive Runtime Strategies via Migratable Objects. In M. Parashar, editor, Advanced Computational Infrastructures for Parallel and Distributed Applications, pages 265--282. Wiley-Interscience, 2009.

[14]

L. V. Kale, G. Zheng, C. W. Lee, and S. Kumar. Scaling applications to massively parallel machines using projections performance analysis tool. In Future Generation Computer Systems Special Issue on: Large-Scale System Performance Modeling and Analysis, volume 22, pages 347--358, February 2006.

Digital Library

[15]

S. Kumar, G. Dozsa, G. Almasi, P. Heidelberger, D. Chen, M. E. Giampapa, B. Michael, A. Faraj, J. Parker, J. Ratterman, B. Smith, and C. J. Archer. The deep computing messaging framework: generalized scalable message passing on the blue gene/p supercomputer. In Proceedings of the 22nd annual international conference on Supercomputing, ICS '08, pages 94--103, New York, NY, USA, 2008. ACM.

Digital Library

[16]

C. Mei, G. Zheng, F. Gioachin, and L. V. Kalé. Optimizing a Parallel Runtime System for Multicore Clusters: A Case Study. In TeraGrid'10, number 10--13, Pittsburgh, PA, USA, August 2010.

Digital Library

[17]

J. C. Phillips, R. Braun, W. Wang, J. Gumbart, E. Tajkhorshid, E. Villa, C. Chipot, R. D. Skeel, L. Kalé, and K. Schulten. Scalable molecular dynamics with NAMD. Journal of Computational Chemistry, 26(16):1781--1802, 2005.

[18]

H. Pourreza and P. Graham. On the programming impact ofmulti-core, multi-processor nodes inmpi clusters. In Proceedings of the 21st International Symposium on High Performance Computing Systems and Applications, pages 1--, Washington, DC, USA, 2007. IEEE Computer Society.

Digital Library

[19]

R. Schulz, B. Lindner, L. Petridis, and J. C. Smith. Scaling of Multimillion-Atom Biological Molecular Dynamics Simulation on a Petascale Supercomputer. Journal of Chemical Theory and Computation, 5(10):2798--2808, Oct. 2009.

[20]

L. Smith and M. Bull. Development of mixed mode MPI/OpenMP applications. Scientific Programming, 9(2--3/2001):83--98, 2001.

Digital Library

[21]

G. Tang, E. F. D'Azevedo, F. Zhang, J. C. Parker, D. B. Watson, and P. M. Jardine. Application of a hybrid mpi/openmp approach for parallel groundwater model calibration using multi-core computers. Comput. Geosci., 36:1451--1460, November 2010.

Digital Library

[22]

G. Zheng, A. Bhatele, E. Meneses, and L. V. Kale. Periodic Hierarchical Load Balancing for Large Supercomputers. International Journal of High Performance Computing Applications (IHHPCA), 2010.

Digital Library

Cited By

Antolínez SJones PPhillips JHadden-Perilla J(2024)AMBERff at Scale: Multimillion-Atom Simulations with AMBER Force Fields in NAMDJournal of Chemical Information and Modeling10.1021/acs.jcim.3c0164864:2(543-554)Online publication date: 4-Jan-2024
https://doi.org/10.1021/acs.jcim.3c01648
Sarthak KWinogradoff DGe YMyong SAksimentiev A(2023)Benchmarking Molecular Dynamics Force Fields for All-Atom Simulations of Biological CondensatesJournal of Chemical Theory and Computation10.1021/acs.jctc.3c0014819:12(3721-3740)Online publication date: 3-May-2023
https://doi.org/10.1021/acs.jctc.3c00148
Anna Victoria Oikawa CFreitas VCastro MPilla L(2020)Adaptive Load Balancing based on Machine Learning for Iterative Parallel Applications2020 28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP)10.1109/PDP50117.2020.00021(94-101)Online publication date: Mar-2020
https://doi.org/10.1109/PDP50117.2020.00021
Show More Cited By

Index Terms

Enabling and scaling biomolecular simulations of 100 million atoms on petascale machines with a multicore-optimized message-driven runtime

Recommendations

A uGNI-based Asynchronous Message-driven Runtime System for Cray Supercomputers with Gemini Interconnect
IPDPS '12: Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium

Gemini, the network for the new Cray XE/XK systems, features low latency, high bandwidth and strong scalability. Its hardware support for remote direct memory access enables efficient implementation of the global address space programming languages. ...
Enabling concurrent multithreaded MPI communication on multicore petascale systems
EuroMPI'10: Proceedings of the 17th European MPI users' group meeting conference on Recent advances in the message passing interface

With the ever-increasing numbers of cores per node on HPC systems, applications are increasingly using threads to exploit the shared memory within a node, combined with MPI across nodes. Achieving high performance when a large number of concurrent ...
Hybrid PGAS runtime support for multicore nodes
PGAS '10: Proceedings of the Fourth Conference on Partitioned Global Address Space Programming Model

With multicore processors as the standard building block for high performance systems, parallel runtime systems need to provide excellent performance on shared memory, distributed memory, and hybrids. Conventional wisdom suggests that threads should be ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

November 2011

866 pages

ISBN:9781450307710

DOI:10.1145/2063384

Conference Chair:
Scott Lathrop
University of Chicago
,
Program Chairs:
Jim Costa
Sandia National Laboratories
,
William Kramer
National Center for Supercomputing Applications

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

SC '11

Sponsor:

SIGARCH
IEEE-CS

SC '11: International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 18, 2011

Washington, Seattle

Acceptance Rates

SC '11 Paper Acceptance Rate 74 of 352 submissions, 21%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
208
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)2

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Antolínez SJones PPhillips JHadden-Perilla J(2024)AMBERff at Scale: Multimillion-Atom Simulations with AMBER Force Fields in NAMDJournal of Chemical Information and Modeling10.1021/acs.jcim.3c0164864:2(543-554)Online publication date: 4-Jan-2024
https://doi.org/10.1021/acs.jcim.3c01648
Sarthak KWinogradoff DGe YMyong SAksimentiev A(2023)Benchmarking Molecular Dynamics Force Fields for All-Atom Simulations of Biological CondensatesJournal of Chemical Theory and Computation10.1021/acs.jctc.3c0014819:12(3721-3740)Online publication date: 3-May-2023
https://doi.org/10.1021/acs.jctc.3c00148
Anna Victoria Oikawa CFreitas VCastro MPilla L(2020)Adaptive Load Balancing based on Machine Learning for Iterative Parallel Applications2020 28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP)10.1109/PDP50117.2020.00021(94-101)Online publication date: Mar-2020
https://doi.org/10.1109/PDP50117.2020.00021
Jung JKobayashi CKasahara KTan CKuroda AMinami KIshiduki SNishiki TInoue HIshikawa YFeig MSugita Y(2020)New parallel computing algorithm of molecular dynamics for extremely huge scale biological systemsJournal of Computational Chemistry10.1002/jcc.2645042:4(231-241)Online publication date: 16-Nov-2020
https://doi.org/10.1002/jcc.26450
Acun BChandrasekar KKale L(2019)Fine-Grained Energy Efficiency Using Per-Core DVFS with an Adaptive Runtime System2019 Tenth International Green and Sustainable Computing Conference (IGSC)10.1109/IGSC48788.2019.8957174(1-8)Online publication date: Oct-2019
https://doi.org/10.1109/IGSC48788.2019.8957174
Freitas VSantana ACastro MPilla L(2019)Distributed Memory Graph Representation for Load Balancing Data: Accelerating Data Structure Generation for Decentralized Scheduling2019 International Conference on High Performance Computing & Simulation (HPCS)10.1109/HPCS48598.2019.9188134(787-794)Online publication date: Jul-2019
https://doi.org/10.1109/HPCS48598.2019.9188134
Chen JAn HYan BLiang WLiu X(2019)Redesign NAMD Molecular Dynamics Non-Bonded Force-Field on Sunway Manycore Processor2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)10.1109/HPCC/SmartCity/DSS.2019.00176(1257-1265)Online publication date: Aug-2019
https://doi.org/10.1109/HPCC/SmartCity/DSS.2019.00176
Phillips JSanielevici S(2018)What You Should Know About NAMD and Charm++ But Were Hoping to IgnoreProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity10.1145/3219104.3219134(1-6)Online publication date: 22-Jul-2018
https://dl.acm.org/doi/10.1145/3219104.3219134
Freitas Vde L. Santana ACastro MPilla L(2018)A Batch Task Migration Approach for Decentralized Global Rescheduling2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)10.1109/CAHPC.2018.8645953(49-56)Online publication date: Sep-2018
https://doi.org/10.1109/CAHPC.2018.8645953
Chong SHam S(2018)Examining a Thermodynamic Order Parameter of Protein FoldingScientific Reports10.1038/s41598-018-25406-88:1Online publication date: 8-May-2018
https://doi.org/10.1038/s41598-018-25406-8
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents