Abstract
Database systems with hybrid data management support, referred to as HTAP or HOAP architectures, are gaining popularity. These first appeared in the relational world, and the CH-benCHmark (CH) was proposed in 2011 to evaluate such relational systems. Today, one finds NoSQL database systems gaining adoption for new applications. In this paper we present CH2, a new benchmark – created with CH as its starting point – aimed at evaluating hybrid data platforms in the document data management world. Like CH, CH2 borrows from and extends both TPC-C and TPC-H. Differences from CH include a document-oriented schema, a data generation scheme that creates a TPC-H-like history, and a “do over” of the CH queries that is more in line with TPC-H. This paper details shortcomings that we uncovered in CH, the design of CH2, and preliminary results from running CH2 against Couchbase Server 7.0 (whose Query and Analytics services provide HOAP support for NoSQL data). The results provide insight into the performance isolation and horizontal scalability properties of Couchbase Server 7.0 as well as demonstrating the efficacy of CH2 for evaluating such platforms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We prefer the term HOAP over HTAP in the context of NoSQL, as it seems less tied to strict ACID transactions and columnar, main-memory technology presumptions.
- 2.
N1QL is short for Non-1NF Query Language.
- 3.
The software artifacts associated with this paper’s benchmark can be found at https://github.com/couchbaselabs/ch2.
- 4.
The reason why we see some NewOrder throughput impact for the smaller 8Q configurations, but apparently without a corresponding NewOrder response time impact, is currently a bit of a performance mystery that running CH2 has revealed. We have several theories and we are currently investigating this behavior in order to further enhance Couchbase Server’s performance isolation and scaling characteristics.
References
451 Research: Hybrid processing enables new use cases (business impact brief) (2018). https://www.intersystems.com/isc-resources/wp-content/uploads/sites/24/Hybrid_Processing_Enables_New_Use_Cases-451Research.pdf. Accessed 19 Oct 2020
Al Hubail, M., et al.: Couchbase analytics: NoETL for scalable NoSQL data analysis. PVLDB 12(12), 2275–2286 (2019)
Borkar, D., et al.: Have your data and query it too: from key-value caching to big data management. In: Proceedings of ACM SIGMOD Conference, pp. 239–251. ACM (2016)
Chamberlin, D.: SQL++ for SQL Users: A Tutorial. Couchbase, Inc. (2018). Amazon.com
Cole, R.L., et al.: The mixed workload CH-benCHmark. In: Proceedings of Fourth International Workshop on Testing Database Systems, DBTest 2011, Athens, Greece, 13 June 2011, p. 8. ACM (2011)
Cooper, B.F., et al.: Benchmarking cloud serving systems with YCSB. In: Proceedings of 1st ACM Symposium on Cloud Computing, SoCC 2010, Indianapolis, Indiana, USA, 10–11 June 2010, pp. 143–154. ACM (2010)
Gray, J. (ed.): The Benchmark Handbook for Database and Transaction Systems, 1st edn. Morgan Kaufmann, Burlington (1991)
Kamsky, A.: Adapting TPC-C benchmark to measure performance of multi-document transactions in MongoDB. PVLDB 12(12), 2254–2262 (2019)
Kemper, A., Neumann, T.: HyPer: a hybrid OLTP&OLAP main memory database system based on virtual memory snapshots. In: 2011 IEEE 27th International Conference on Data Engineering, pp. 195–206 (2011)
Lahiri, T., et al.: Oracle database in-memory: a dual format in-memory database. In: 2015 IEEE 31st International Conference on Data Engineering, pp. 1253–1258 (2015)
Larson, P., et al.: Real-time analytical processing with SQL server. PVLDB 8(12), 1740–1751 (2015)
May, N., Böhm, A., Lehner, W.: SAP HANA - the evolution of an in-memory DBMS from pure OLAP processing towards mixed workloads. In: Proceedings of BTW 2017, 17. Fachtagung des GI-Fachber. DBIS, März 2017, Stuttgart, Germany (2017)
Özsu, M.T., Valduriez, P.: Principles of Distributed Database Systems, 4th edn. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-26253-2
Pirzadeh, P., Carey, M., Westmann, T.: BigFUN: a performance study of big data management system functionality. In: 2015 IEEE International Conference on Big Data, pp. 507–514 (2015)
Pirzadeh, P., Carey, M., Westmann, T.: A performance study of big data analytics platforms. In: 2017 IEEE International Conference on Big Data, pp. 2911–2920 (2017)
Pöss, M., Floyd, C.: New TPC benchmarks for decision support and web commerce. SIGMOD Rec. 29(4), 64–71 (2000)
Pöss, M., et al.: TPC-DS, taking decision support benchmarking to the next level. In: Proceedings of ACM SIGMOD Conference, pp. 582–587. ACM (2002)
Raab, F.: TPC-C - the standard benchmark for online transaction processing (OLTP). In: Gray, J. (ed.) The Benchmark Handbook for Database and Transaction Systems, 2nd edn. Morgan Kaufmann (1993)
Raman, V., et al.: DB2 with BLU acceleration: so much more than just a column store. PVLDB 6(11), 1080–1091 (2013)
Raza, A., et al.: Adaptive HTAP through elastic resource scheduling. In: Proceedings of ACM SIGMOD Conference, pp. 2043–2054. ACM (2020)
Sadalage, P.J., Fowler, M.: NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. Addison-Wesley, Upper Saddle River (2013)
Shasha, D.E.: Database Tuning - A Principled Approach. Prentice-Hall, Hoboken (1992)
Tian, Y., Carey, M., Maxon, I.: Benchmarking HOAP for scalable document data management: a first step. In: 2020 IEEE International Conference on Big Data, pp. 2833–2842 (2020)
Wikipedia contributors: Hybrid transactional/analytical processing – Wikipedia, the free encyclopedia (2020). https://en.wikipedia.org/w/index.php?title=Hybrid_transactional/analytical_processing&oldid=981969658. Accessed 19 Oct 2020
Acknowledgments
The authors wish to thank the Couchbase Query Service team, especially Sitaram Vemulapalli and Kamini Jagtiani, for assisting us with the new 7.0 N1QL transaction support, and Michael Blow and Ian Maxon from the Couchbase Analytics Service team, for invaluable assistance in setting up the AWS clusters used for the experiments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Carey, M., Lychagin, D., Muralikrishna, M., Sarathy, V., Westmann, T. (2022). CH2: A Hybrid Operational/Analytical Processing Benchmark for NoSQL. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2021. Lecture Notes in Computer Science(), vol 13169. Springer, Cham. https://doi.org/10.1007/978-3-030-94437-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-94437-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-94436-0
Online ISBN: 978-3-030-94437-7
eBook Packages: Computer ScienceComputer Science (R0)