keynote

Performance Engineering and Database Development at MongoDB

Author:

David DalyAuthors Info & Claims

ICPE '21: Companion of the ACM/SPEC International Conference on Performance Engineering

Page 129

https://doi.org/10.1145/3447545.3451199

Published: 19 April 2021 Publication History

Get Access

Abstract

Performance and the related properties of stability and resilience are essential to MongoDB. We have invested heavily in these areas: involving all development engineers in aspects of performance, building a team of specialized performance engineers to understand issues that do not fit neatly within the scope of individual development teams, and dedicating multiple teams to develop and support tools for performance testing and analysis.

We have built an automated infrastructure for performance testing that is integrated with our continuous integration system. Performance tests routinely run against our development branch in order to identify changes in performance as early as possible. We have invested heavily to ensure both that results are low noise and reproducible and that we can detect when performance changes. We continue to invest to make the system better and to make it easier to add new workloads.

All development engineers are expected to interact with our performance system: investigating performance changes, fixing regressions, and adding new performance tests. We also expect performance to be considered at project design time. The project design should ensure that there is appropriate performance coverage for the project, which may require repurposing existing tests or adding new ones.

Not all performance issues are specific to a team or software module. Some issues emerge from the interaction of multiple modules or interaction with external systems or software. To attack these larger problems, we have a dedicated performance team. Our performance team is responsible for investigating these more complex issues, identifying high value areas for improvement, as well as helping guide the development engineers with their performance tests.

We have experience both hiring and training engineers for the performance engineering skills needed to ship a performant database system. In this talk we will cover the skills needed for our performance activities and which skills, if added to undergraduate curricula, would help us the most. We will address the skills we would like all development engineers to have, as well as those for our dedicated performance team.

References

[1]

2020. Genny workload generator. https://github.com/mongodb/genny original-date: 2018-02-12T19:23:44Z.

Google Scholar

[2]

David Daly. 2021. Creating a Virtuous Cycle in Performance Testing at MongoDB. In Proceedings of the 2021 ACM/SPEC International Conference on Performance Engineering (ICPE '21), April 19--23, 2021, Virtual Event, France. ACM. https://doi.org/10.1145/3427921.3450234

Digital Library

Google Scholar

[3]

David Daly, William Brown, Henrik Ingo, Jim O'Leary, and David Bradford. 2020. The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System. In Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE '20). Association for Computing Machinery, Edmonton AB, Canada, 67--75. https://doi.org/10.1145/3358960.3375791

Digital Library

Google Scholar

[4]

Kyle Erf. 2016. Evergreen Continuous Integration: Why We Reinvented The Wheel. https://engineering.mongodb.com/post/evergreen-continuous-integration-why-we-reinvented-the-wheel

Google Scholar

[5]

Henrik Ingo and David Daly. 2019. Reducing variability in performance tests on EC2: Setup and Key Results. https://engineering.mongodb.com/post/reducing-variability-in-performance-tests-on-ec2-setup-and-key-results

Google Scholar

[6]

Henrik Ingo and David Daly. 2020. Automated system performance testing at MongoDB. In Proceedings of the workshop on Testing Database Systems (DBTest '20). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/3395032.3395323

Digital Library

Google Scholar

Cited By

View all

Miftakhov EIvanov D(2024)Utilizing Cloud Technologies in the Investigation of Physicochemical Processes2024 International Russian Smart Industry Conference (SmartIndustryCon)10.1109/SmartIndustryCon61328.2024.10515726(819-823)Online publication date: 25-Mar-2024
https://doi.org/10.1109/SmartIndustryCon61328.2024.10515726
Miftakhov EIvanov DMustafina S(2023)Creation of a Cloud Saas Service for a Comprehensive Study of Physical and Chemical Processes2023 5th International Conference on Control Systems, Mathematical Modeling, Automation and Energy Efficiency (SUMMA)10.1109/SUMMA60232.2023.10349413(495-498)Online publication date: 8-Nov-2023
https://doi.org/10.1109/SUMMA60232.2023.10349413
Bach TAndrzejak ASeo CBierstedt CLemke CRitter DHwang DSheshi ESchabernack FRenkes FGaumnitz GMartens JHoemke LFelderer MRudolf MJambigi NMay NJoy RScheja RSchwedes SSeibel SSeifert SHaas SKraft SKroll TScheuer TLehner W(2022)Testing Very Large Database Management Systems: The Case of SAP HANADatenbank-Spektrum10.1007/s13222-022-00426-x22:3(195-215)Online publication date: 24-Nov-2022
https://doi.org/10.1007/s13222-022-00426-x
Show More Cited By

Index Terms

Performance Engineering and Database Development at MongoDB

Recommendations

Understanding and Improving Software Performance at MongoDB
ICPE '23 Companion: Companion of the 2023 ACM/SPEC International Conference on Performance Engineering

It is important for developers to understand the performance of a software project as they develop new features, fix bugs, and try to generally improve the product. At MongoDB we have invested in building a performance infrastructure to support our ...
Creating a Virtuous Cycle in Performance Testing at MongoDB
ICPE '21: Proceedings of the ACM/SPEC International Conference on Performance Engineering

It is important to detect changes in software performance during development in order to avoid performance decreasing release to release or dealing with costly delays at release time. Performance testing is part of the development process at MongoDB, ...
The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System
ICPE '20: Proceedings of the ACM/SPEC International Conference on Performance Engineering

We describe our process for automatic detection of performance changes for a software product in the presence of noise. A large collection of tests run periodically as changes to our software product are committed to our source repository, and we would ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ICPE '21: Companion of the ACM/SPEC International Conference on Performance Engineering

April 2021

198 pages

ISBN:9781450383318

DOI:10.1145/3447545

General Chairs:
Johann Bourcier
University of Rennes 1, France
,
Zhen Ming (Jack) Jiang
York University, Canada
,
Program Chairs:
Cor-Paul Bezemer
University of Alberta, Canada
,
Vittorio Cortellessa
University of L'Aquila, Italy

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2021

Check for updates

Author Tags

Qualifiers

Keynote

Conference

ICPE '21

Sponsor:

ICPE '21: ACM/SPEC International Conference on Performance Engineering

April 19 - 23, 2021

Virtual Event, France

Acceptance Rates

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
136
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)2

Reflects downloads up to 18 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Miftakhov EIvanov D(2024)Utilizing Cloud Technologies in the Investigation of Physicochemical Processes2024 International Russian Smart Industry Conference (SmartIndustryCon)10.1109/SmartIndustryCon61328.2024.10515726(819-823)Online publication date: 25-Mar-2024
https://doi.org/10.1109/SmartIndustryCon61328.2024.10515726
Miftakhov EIvanov DMustafina S(2023)Creation of a Cloud Saas Service for a Comprehensive Study of Physical and Chemical Processes2023 5th International Conference on Control Systems, Mathematical Modeling, Automation and Energy Efficiency (SUMMA)10.1109/SUMMA60232.2023.10349413(495-498)Online publication date: 8-Nov-2023
https://doi.org/10.1109/SUMMA60232.2023.10349413
Bach TAndrzejak ASeo CBierstedt CLemke CRitter DHwang DSheshi ESchabernack FRenkes FGaumnitz GMartens JHoemke LFelderer MRudolf MJambigi NMay NJoy RScheja RSchwedes SSeibel SSeifert SHaas SKraft SKroll TScheuer TLehner W(2022)Testing Very Large Database Management Systems: The Case of SAP HANADatenbank-Spektrum10.1007/s13222-022-00426-x22:3(195-215)Online publication date: 24-Nov-2022
https://doi.org/10.1007/s13222-022-00426-x
Pustulka Evon Arx Sde Espona L(2022)Building a NoSQL ERPProceedings of Seventh International Congress on Information and Communication Technology10.1007/978-981-19-1610-6_59(671-680)Online publication date: 27-Jul-2022
https://doi.org/10.1007/978-981-19-1610-6_59

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Understanding and Improving Software Performance at MongoDB

Creating a Virtuous Cycle in Performance Testing at MongoDB

The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System