abstract

Open Problems in Stream Processing: A Call To Action

Author:

Tyler AkidauAuthors Info & Claims

DEBS '19: Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems

Page 4

https://doi.org/10.1145/3328905.3338223

Published: 24 June 2019 Publication History

Get Access

Abstract

In the last four years, stream processing has gone from niche to mainstream, with real-time data processing systems gaining traction in not only fast-moving startups, but also their more skeptical and cautious enterprise brethren. In light of such pervasive adoption, is it safe to say we've we finally reached the point where stream processing is a solved commodity? Are we done and ready to move on to the next big thing?

In this talk, I will argue that the answer to those questions is conclusively "no": stream processing as a field of research is alive and well. In fact, as streaming systems evolve to look more and more alike, the need for active exploration of new ideas is all the more pressing. And though streaming systems are more capable and robust than ever, they remain in many ways difficult to use, difficult to maintain, and difficult to understand. But we can change that.

I don't claim to have all the answers; no one does. But I do have a few ideas of where we can start. And by sharing my thoughts on some of the more interesting open problems in stream processing, and encouraging others to share theirs, I'm hoping that we as a research community can work together to help move the needle just a little bit further.

Cited By

View all

Henning SHasselbring W(2021)TheodoliteBig Data Research10.1016/j.bdr.2021.10020925:COnline publication date: 15-Jul-2021
https://dl.acm.org/doi/10.1016/j.bdr.2021.100209
Stylianopoulos CAlmgren MLandsiedel OPapatriantafilou M(2020)Multiple pattern matching for network security applicationsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2019.10.011137:C(34-52)Online publication date: 1-Mar-2020
https://dl.acm.org/doi/10.1016/j.jpdc.2019.10.011

Recommendations

Dual-Paradigm Stream Processing
ICPP '18: Proceedings of the 47th International Conference on Parallel Processing

Existing stream processing frameworks operate either under data stream paradigm processing data record by record to favor low latency, or under operation stream paradigm processing data in micro-batches to desire high throughput. For complex and mutable ...
Stream processing coalgebraically

We study various operations for splitting, partitioning, projecting and merging streams of data. These operations are motivated by their use in dataflow programming and stream processing languages. We use the framework of stream calculus and stream ...
Expressive and scalable event stream processing

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

DEBS '19: Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems

June 2019

291 pages

ISBN:9781450367943

DOI:10.1145/3328905

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 June 2019

Check for updates

Qualifiers

Abstract
Research
Refereed limited

Conference

DEBS '19

Sponsor:

DEBS '19: The 13th ACM International Conference on Distributed and Event-based Systems

June 24 - 28, 2019

Darmstadt, Germany

Acceptance Rates

DEBS '19 Paper Acceptance Rate 13 of 47 submissions, 28%;

Overall Acceptance Rate 145 of 583 submissions, 25%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
290
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Henning SHasselbring W(2021)TheodoliteBig Data Research10.1016/j.bdr.2021.10020925:COnline publication date: 15-Jul-2021
https://dl.acm.org/doi/10.1016/j.bdr.2021.100209
Stylianopoulos CAlmgren MLandsiedel OPapatriantafilou M(2020)Multiple pattern matching for network security applicationsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2019.10.011137:C(34-52)Online publication date: 1-Mar-2020
https://dl.acm.org/doi/10.1016/j.jpdc.2019.10.011

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Dual-Paradigm Stream Processing

Stream processing coalgebraically

Expressive and scalable event stream processing