Cited By
View all- Frischbier SPaic MEchler ARoth C(2019)Managing the Complexity of Processing Financial Data at Scale - An Experience ReportComplex Systems Design & Management10.1007/978-3-030-34843-4_2(14-26)Online publication date: 27-Nov-2019
Benchmarking Financial Data Feed Systems
Authors: 
Manuel Coenen, Christoph Wagner, Alexander Echler, Sebastian FrischbierAuthors Info & Claims
Pages 252 - 253
Abstract
Data-driven solutions for the investment industry require event-based backend systems to process high-volume financial data feeds with low latency, high throughput, and guaranteed delivery modes.
At vwd we process an average of 18 billion incoming event notifications from 500+ data sources for 30 million symbols per day and peak rates of 1+ million notifications per second using custom-built platforms that keep audit logs of every event.
We currently assess modern open source event-processing platforms such as Kafka, NATS, Redis, Flink or Storm for the use in our ticker plant to reduce the maintenance effort for cross-cutting concerns and leverage hybrid deployment models. For comparability and repeatability we benchmark candidates with a standardized workload we derived from our real data feeds.
We have enhanced an existing light-weight open source benchmarking tool in its processing, logging, and reporting capabilities to cope with our workloads. The resulting tool wrench can simulate workloads or replay snapshots in volume and dynamics like those we process in our ticker plant. We provide the tool as open source.
As part of ongoing work we contribute details on (a) our workload and requirements for benchmarking candidate platforms for financial feed processing; (b) the current state of the tool wrench.
References
[1]
Andre L.S. Gradvohl. 2016. Investigating Metrics to Build a Benchmark Tool for Complex Event Processing Systems. In IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW '16). IEEE, 143--147.
[2]
Marcelo R.N. Mendes, Pedro Bizarro, and Paulo Marques. 2013. Towards a Standard Event Processing Benchmark. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE '13). ACM, 307--310.
[3]
Anshu Shukla and Yogesh Simmhan. 2017. Benchmarking Distributed Stream Processing Platforms for IoT Applications. In Performance Evaluation and Benchmarking. Traditional - Big Data - Internet of Things (TPCTC '16). Springer, 90--106.
[4]
Tyler Treat. 2017. bench - A generic latency benchmarking library. https://github.com/tylertreat/bench. {Online; accessed 2019-04-17}.
[5]
vwdsrc. 2019. wrench - Workload-optimized & Reengineered bench. https://github.com/vwdsrc/wrench. {Online; accessed 2019-04-17}.
Index Terms
- Benchmarking Financial Data Feed Systems
Recommendations
Deriving a realistic workload model to simulate high-volume financial data feeds for performance benchmarking
DEBS '22: Proceedings of the 16th ACM International Conference on Distributed and Event-Based SystemsProcessing financial market data at scale and in real-time poses a set of unique challenges to event-driven architectures due to the volume, variety, velocity, and veracity of the enclosed information on top of other constraints. Reproducible stress ...
A Real-World Distributed Infrastructure for Processing Financial Data at Scale
DEBS '19: Proceedings of the 13th ACM International Conference on Distributed and Event-based SystemsFinancial markets are event- and data-driven to an extremely high degree. For making decisions and triggering actions stakeholders require notifications about significant events and reliable background information that meet their individual requirements ...
Issues in big data testing and benchmarking
DBTest '13: Proceedings of the Sixth International Workshop on Testing Database SystemsThe academic community and industry are currently researching and building next generation data management systems. These systems are designed to analyze data sets of high volume with high data ingest rates and short response times executing complex ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2019
291 pages
ISBN:9781450367943
DOI:10.1145/3328905
Copyright © 2019 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 June 2019
Check for updates
Author Tags
Qualifiers
- Poster
- Research
- Refereed limited
Conference
DEBS '19: The 13th ACM International Conference on Distributed and Event-based Systems
June 24 - 28, 2019
Darmstadt, Germany
Acceptance Rates
DEBS '19 Paper Acceptance Rate 13 of 47 submissions, 28%;
Overall Acceptance Rate 145 of 583 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 147Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)2
Reflects downloads up to 14 Jan 2025
Other Metrics
Citations
Cited By
View all- Frischbier SPaic MEchler ARoth C(2019)Managing the Complexity of Processing Financial Data at Scale - An Experience ReportComplex Systems Design & Management10.1007/978-3-030-34843-4_2(14-26)Online publication date: 27-Nov-2019
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in