More Web Proxy on the site http://driver.im/

research-article

Open access

An early evaluation of Intel's optane DC persistent memory module and its impact on high-performance scientific applications

Authors:

Michèle Weiland,

Tiago Quintino,

Olivier Iffrig,

Christian Herold,

Antonino Bonanni,

Adrian Jackson,

Mark ParsonsAuthors Info & Claims

SC '19: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 76, Pages 1 - 19

https://doi.org/10.1145/3295500.3356159

Published: 17 November 2019 Publication History

Abstract

Memory and I/O performance bottlenecks in supercomputing simulations are two key challenges that must be addressed on the road to Exascale. The new byte-addressable persistent non-volatile memory technology from Intel, DCPMM, promises to be an exciting opportunity to break with the status quo, with unprecedented levels of capacity at near-DRAM speeds. Here, we explore the potential of DCPMM in the context of two high-performance scientific applications in terms of outright performance, efficiency and usability for both its Memory and App Direct modes. In Memory mode, we show equivalent performance and better efficiency for a CASTEP simulation that is limited by memory capacity on conventional DRAM-only systems without any changes to the application. For IFS, we demonstrate that a distributed object-store over NVRAM reduces the data contention created in weather forecasting data producer-consumer workflows. In addition, we also present the achievable memory bandwidth performance using STREAM.

References

[1]

2013. Intelligent Platform Management Interface Specification v2.0 rev. 1.1. Retrieved April 9, 2019 from https://www.intel.co.uk/content/dam/www/public/us/en/documents/product-briefs/ipmi-second-gen-interface-spec-v2-rev1-1.pdf

[2]

2015. perf: Linux profiling with performance counters. Retrieved May 22, 2019 from http://perf.wiki.kernel.org

[3]

2017. CASTEP DNA benchmark. Retrieved March 25, 2019 from http://www.castep.org/CASTEP/DNA

[4]

2017. CASTEP TiN benchmark. Retrieved April 5, 2019 from http://www.castep.org/CASTEP/TiN

[5]

2019. ECMWF Datasets. Retrieved April 10, 2019 from https://www.ecmwf.int/en/forecasts/datasets

[6]

2019. ECMWF website. Retrieved April 10, 2019 from https://www.ecmwf.int

[7]

2019. Intel Announces Broadest Product Portfolio for Moving, Storing and Processing Data. Retrieved April 8, 2019 from http://newsroom.intel.com/news-releases/intel-data-centric-launch

[8]

2019. IPMCTL. Retrieved April 4, 2019 from https://github.com/intel/ipmctl

[9]

2019. NDCTL - Utility library for managing the libnvdimm (non-volatile memory device) sub-system in the Linux kernel. Retrieved May 24, 2019 from https://github.com/pmem/ndctl

[10]

2019. NVM Programming Model (NPM), Version 1.2. Retrieved April 4, 2019 from https://www.snia.org/sites/default/files/technical_work/final/NVMProgrammingModel_v1.2.pdf

[11]

2019. Persistent Memory Development Kit. Retrieved April 3, 2019 from http://pmem.io/pmdk/

[12]

2019. SLURM accounting. Retrieved April 5, 2019 from https://slurm.schedmd.com/sacct.html

[13]

2019. U.S. Department of Energy and Intel to deliver first exascale supercomputer. Retrieved April 8, 2019 from http://www.anl.gov/article/us-department-of-energy-and-intel-to-deliver-first-exascale-supercomputer

[14]

Sadaf R. Alam, Hussein N. El-Harake, Kristopher Howard, Neil Stringfellow, and Fabio Verzelloni. 2011. Parallel I/O and the Metadata Wall. In Proceedings of the Sixth Workshop on Parallel Data Storage (PDSW '11). ACM, New York, NY, USA, 13--18.

Digital Library

[15]

Julian Borrill, Leonid Oliker, John Shalf, and Hongzhang Shan. 2007. Investigation of Leading HPC I/O Performance Using a Scientific-application Derived Benchmark. In Proceedings of the 2007 ACM/IEEE Conference on Super-computing (SC '07). ACM, New York, NY, USA, Article 10, 12 pages.

Digital Library

[16]

Adrian M. Caulfield, Joel Coburn, Todor Mollov, Arup De, Ameen Akel, Jiahua He, Arun Jagatheesan, Rajesh K. Gupta, Allan Snavely, and Steven Swanson. 2010. Understanding the Impact of Emerging Non-Volatile Memories on High-Performance, IO-Intensive Computing. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC '10). IEEE Computer Society, Washington, DC, USA, 1--11.

Digital Library

[17]

S. J. Clark, M. D. Segall, C. J. Pickard, P. J. Hasnip, M. J. Probert, K. Refson, and M.C. Payne. 2005. First principles methods using CASTEP. Z. Kristall. 220 (2005), 567--570.

[18]

ECMWF. 2015. ECMWF Strategy 2016-2015, The strength of a common goal. http://www.ecmwf.int/sites/default/files/ECMWF_Strategy_2016-2025.pdf.

[19]

Pradeep Fernando, Ada Gavrilovska, Sudarsun Kannan, and Greg Eisenhauer. 2018. NVStream: Accelerating HPC Workflows with NVRAM-based Transport for Streaming Objects. In Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing (HPDC '18). ACM, New York, NY, USA, 231--242.

Digital Library

[20]

P. Fernando, S. Kannan, A. Gavrilovska, and K. Schwan. 2016. Phoenix: Memory Speed HPC I/O with NVM. In 2016 IEEE 23rd International Conference on High Performance Computing (HiPC). 121--131.

[21]

Edward Higgins, Matt Probert, Phil Hasnip, Keith Refson, and Ian Bush. 2015. Hybrid OpenMP and MPI within the CASTEP code. ARCHER eCSE Technical Report (2015). http://www.archer.ac.uk/community/eCSE/eCSE01-017/eCSE01-017_Final_Report_technical.pdf

[22]

P. Hohenberg and W. Kohn. 1964. Inhomogeneous electron gas. Phys. Rev. 136 (1964), B864--B871.

[23]

Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R. Dulloor, Jishen Zhao, and Steven Swanson. 2019. Basic Performance Measurements of the Intel Optane DC Persistent Memory Module. CoRR abs/1903.05714 (2019). arXiv:1903.05714 http://arxiv.org/abs/1903.05714

[24]

J. Kim and H. Bahn. 2018. Accelerating Storage Performance with NVRAM by Considering Application's I/O Characteristics. In 2018 IEEE International Conference on Big Data and Smart Computing (BigComp). 383--389.

[25]

Andreas Knüpfer, Holger Brunst, Jens Doleschal, Matthias Jurenz, Matthias Lieber, Holger Mickler, Matthias S. Müller, and Wolfgang E. Nagel. 2008. The Vampir Performance Analysis Tool-Set. In Tools for High Performance Computing, Michael Resch, Rainer Keller, Valentin Himmler, Bettina Krammer, and Alexander Schulz (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 139--155.

[26]

Andreas Knüpfer, Christian Rössel, Dieter an Mey, Scott Biersdorff, Kai Diethelm, Dominic Eschweiler, Markus Geimer, Michael Gerndt, Daniel Lorenz, Allen Malony, Wolfgang E. Nagel, Yury Oleynik, Peter Philippen, Pavel Saviankou, Dirk Schmidl, Sameer Shende, Ronny Tschüter, Michael Wagner, Bert Wesarg, and Felix Wolf. 2012. Score-P: A Joint Performance Measurement Run-Time Infrastructure for Periscope,Scalasca, TAU, and Vampir. In Tools for High Performance Computing 2011, Holger Brunst, Matthias S. Müller, Wolfgang E. Nagel, and Michael M. Resch (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 79--91.

[27]

W. Kohn and L. J. Sham. 1965. Self-consistent equations including exchange and correlation effects. Phys. Rev. 140 (1965), A1133--A1138.

[28]

N. Liu, J. Cope, P. Carns, C. Carothers, R. Ross, G. Grider, A. Crume, and C. Maltzahn. 2012. On the role of burst buffers in leadership-class storage systems. In 012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST). 1--11.

[29]

S. Malardel, N. Wedi, W. Deconinck, M. Diamantakis, C. Kuehnlein, G. Mozdzynski, M. Hamrud, and P. Smolarkiewicz. [n. d.]. A new grid for the IFS. ECMWF Newsletter 146 ([n. d.]), 23--28.

[30]

J. D. McCalpin. 1995. Memory Bandwidth and Machine Balance in Current High Performance Computers. IEEE Technical Committee on Computer Architecture (TCCA) Newsletter (Dec 1995).

[31]

M. C. Payne, M. P. Teter, D. C. Allan, T.A. Arias, and J. D. Joannopoulos. 1992. Iterative minimization techniques for ab initio total-energy calculations - molecular-dynamics and conjugate gradients. Rev. Mod. Phys. 64 (1992), 1045--1097.

[32]

Andy Rudoff. 2017. Persistent Memory: The Value to HPC and the Challenges. In Proceedings of the Workshop on Memory Centric Programming for HPC (MCHPC'17). ACM, New York, NY, USA, 7--10.

Digital Library

[33]

Simon D. Smart, Tiago Quintino, and Baudouin Raoult. 2017. A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate. In Submitted to Proceedings of the Platform for Advanced Scientific Computing Conference (PASC '19). ACM, New York, NY, USA.

[34]

Simon D. Smart, Tiago Quintino, and Baudouin Raoult. 2017. A Scalable Object Store for Meteorological and Climate Data. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC '17). ACM, New York, NY, USA, Article 13, 8 pages.

Digital Library

[35]

David Thaler and Chinya Ravishankar. 1996. A Name-Based Mapping Scheme for Rendezvous. http://www.eecs.umich.edu/techreports/cse/96/CSE-TR-316-96.pdf.

[36]

Ananta Tiwari, Anthony Gamst, Michael A. Laurenzano, Martin Schulz, and Laura Carrington. 2014. Modeling the Impact of Reduced Memory Bandwidth on HPC Applications. In Euro-Par 2014 Parallel Processing, Fernando Silva, Inês Dutra, and Vítor Santos Costa (Eds.). Springer International Publishing, Cham, 63--74.

[37]

Michele Weiland, Adrian Jackson, Nick Johnson, and Mark Parsons. 2018. Exploiting the Performance Benefits of Storage Class Memory for HPC and HPDA Workflows. Supercomputing Frontiers and Innovations 5, 1 (2018). http://superfri.org/superfri/article/view/164

[38]

World Meteorological Organization. 2015. GRIB Format. http://www.wmo.int/pages/prog/www/WMOCodes/WMO306_vI2/Publications/2015editionUP2018/WMO306_vI2_en_ONLINE.pdf.

Cited By

Sarmany DValentini MMaciel PGeier PSmart SAguridan RHawkes JQuintino TEvans KSchenk O(2024)MultIO: A Framework for Message-Driven Data Routing For Weather and Climate SimulationsProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3659914.3659938(1-12)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3659914.3659938
Brashear WReddy VBaum SChakravorty DDang FPerez LLiu H(2024)Exploring the Viability of Composable Architectures to Overcome Memory Limitations in High Performance Computing WorkflowsPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670620(1-4)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670620
Cai MShen JTang BHuang HYe B(2024)Exploiting Flat Namespace to Improve File System Metadata Performance on Ultra-Fast, Byte-Addressable NVMsACM Transactions on Storage10.1145/362067320:1(1-47)Online publication date: 30-Jan-2024
https://dl.acm.org/doi/10.1145/3620673
Show More Cited By

Index Terms

An early evaluation of Intel's optane DC persistent memory module and its impact on high-performance scientific applications
1. Computer systems organization
  1. Architectures
    1. Parallel architectures
      1. Multicore architectures
2. Hardware
  1. Emerging technologies
    1. Memory and dense storage

Recommendations

Performance characterization of a DRAM-NVM hybrid memory architecture for HPC applications using intel optane DC persistent memory modules
MEMSYS '19: Proceedings of the International Symposium on Memory Systems

Non-volatile, byte-addressable memory (NVM) has been introduced by Intel in the form of NVDIMMs named Intel® Optane™ DC PMM. This memory module has the ability to persist the data stored in it without the need for power. This expands the memory ...
System evaluation of the Intel optane byte-addressable NVM
MEMSYS '19: Proceedings of the International Symposium on Memory Systems

Byte-addressable non-volatile memory (NVM) features high density, DRAM comparable performance, and persistence. These characteristics position NVM as a promising new tier in the memory hierarchy. Nevertheless, NVM has asymmetric read and write ...
A read-efficient and write-optimized hash table for Intel Optane DC Persistent Memory
Abstract
Emerging non-volatile memory technologies are driving the next generation of storage systems and durable data structures. Among them, many hash table proposals employ NVM as the storage layer for both fast access and efficient persistence. Most ...
Highlights
- A cached table-inclined read mechanism speeds up search requests.
- A log-free atomic write mechanism minimizes overhead for write requests.
- An efficient shadowing synchronization mechanism enables fast data synchronization.
- A ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '19: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2019

1921 pages

ISBN:9781450362290

DOI:10.1145/3295500

General Chair:
Michela Taufer,
Program Chairs:
Pavan Balaji,
Antonio J. Peña

Copyright © 2019 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

H2020 Future and Emerging Technologies

Conference

SC '19

Sponsor:

SIGHPC

SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis

November 17 - 19, 2019

Colorado, Denver

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
3,371
Total Downloads

Downloads (Last 12 months)953
Downloads (Last 6 weeks)47

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sarmany DValentini MMaciel PGeier PSmart SAguridan RHawkes JQuintino TEvans KSchenk O(2024)MultIO: A Framework for Message-Driven Data Routing For Weather and Climate SimulationsProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3659914.3659938(1-12)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3659914.3659938
Brashear WReddy VBaum SChakravorty DDang FPerez LLiu H(2024)Exploring the Viability of Composable Architectures to Overcome Memory Limitations in High Performance Computing WorkflowsPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670620(1-4)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670620
Cai MShen JTang BHuang HYe B(2024)Exploiting Flat Namespace to Improve File System Metadata Performance on Ultra-Fast, Byte-Addressable NVMsACM Transactions on Storage10.1145/362067320:1(1-47)Online publication date: 30-Jan-2024
https://dl.acm.org/doi/10.1145/3620673
Chen ZHu DChe WSun JChen H(2024)A quantitative evaluation of persistent memory hash indexesThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-023-00812-133:2(375-397)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s00778-023-00812-1
Kim SShim JLee EJeong SKang IKim JNaor DGoel A(2023)NVMeVirtProceedings of the 21st USENIX Conference on File and Storage Technologies10.5555/3585938.3585962(379-393)Online publication date: 21-Feb-2023
https://dl.acm.org/doi/10.5555/3585938.3585962
Cai MShen JYuan YQu ZYe B(2023)BonsaiKV: Towards Fast, Scalable, and Persistent Key-Value Stores with Tiered, Heterogeneous Memory SystemProceedings of the VLDB Endowment10.14778/3636218.363622817:4(726-739)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.14778/3636218.3636228
Koutsoukos DBhartia RFriedman MKlimovic AAlonso G(2023)NVM: Is it Not Very Meaningful for Databases?Proceedings of the VLDB Endowment10.14778/3603581.360358616:10(2444-2457)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.14778/3603581.3603586
Kim SShim JLee EJeong SKang IKim J(2023)Empowering Storage Systems Research with NVMeVirt: A Comprehensive NVMe Device EmulatorACM Transactions on Storage10.1145/362500619:4(1-26)Online publication date: 21-Sep-2023
https://dl.acm.org/doi/10.1145/3625006
Fridman YMutalik Desai SSingh NWillhalm TOren G(2023)CXL Memory as Persistent Memory for Disaggregated HPC: A Practical ApproachProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624175(983-994)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624175
Ye CXu YShen XSha YLiao XJin HSolihin YAamodt TJerger NSwift M(2023)SpecPMT: Speculative Logging for Resolving Crash Consistency Overhead of Persistent MemoryProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3575693.3575696(762-777)Online publication date: 27-Jan-2023
https://dl.acm.org/doi/10.1145/3575693.3575696
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents