demonstration

Flood: elastic streaming MapReduce

Authors:

David Alves,

Pedro Bizarro,

Paulo MarquesAuthors Info & Claims

DEBS '10: Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems

Pages 113 - 114

https://doi.org/10.1145/1827418.1827445

Published: 12 July 2010 Publication History

Get Access

Abstract

Distributed data stream processing (DSP) is used to analyze information and raise alarms in business-critical scenarios such as financial fraud-detection, clickstream processing, network security, traffic control, or real-time KPI computations. Processing this information efficiently is very challenging because the nature of continuous streaming sources is varying in nature: often the amount of data and processing changes with time of day and day of week and frequently has unexpected spikes. Thus, the result is that most DSP computations are either over-provisioned, introducing increased cost and wasted energy, or are under-provisioned and, either incur in performance degradation or denial-of-service, or have to resort to load shedding.

We demonstrate Flood, a scalable, elastic DSP engine that addresses these problems. By using a scalable computing model, MapReduce, and adequately monitoring running computations our system is able to decide, in runtime, if there is a lack or a surplus of resources. Flood then acts autonomically by requesting or releasing computing nodes, without losing tuples or redoing computation, at the same time making sure that latency and throughput requirements are guaranteed.

References

[1]

Dean, J and Ghemawat, S. 2004. MapReduce: Simplified Data Processing on Large Clusters. Usenix Annual Technical Conference.

Digital Library

Google Scholar

[2]

Logothetis, D and Yocum, K. 2008. Ad-hoc data processing in the cloud. Proceedings of the VLDB Endowment.

Digital Library

Google Scholar

[3]

Yang, Hung-chih, Dasdan, Ali, Hsiao, Ruey-Lung, and Parker, D. 2007. Map-reduce-merge: simplified relational data processing on large clusters. Proceedings of the 2007 ACM SIGMOD international conference on Management of data.

Digital Library

Google Scholar

[4]

Arpaci-Dusseau, R, Anderson, E, and Treuhaft, N. 1999. Cluster I/O with River: Making the fast case common. Proceedings of the sixth workshop on I/O in parallel and distributed systems.

Digital Library

Google Scholar

[5]

Shah, M. A, Hellerstein, J. M, Sirish Chandrasekaran, and Franklin, M. J. 2003. Flux: an adaptive partitioning operator for continuous query systems. Proceedings of the 19th International Conference on Data Engineering

Google Scholar

Cited By

View all

Stella GLoguinov D(2022)On High-Latency Bowtie Data Streaming2022 IEEE International Conference on Big Data (Big Data)10.1109/BigData55660.2022.10020231(75-84)Online publication date: 17-Dec-2022
https://doi.org/10.1109/BigData55660.2022.10020231
Madsen KZhou YSu LGal AWeidlich MKalogeraki VVenkasubramanian N(2016)EnormProceedings of the 10th ACM International Conference on Distributed and Event-based Systems10.1145/2933267.2933315(37-48)Online publication date: 13-Jun-2016
https://dl.acm.org/doi/10.1145/2933267.2933315
Madsen KZhou YBailey JMoffat AAggarwal Cde Rijke MKumar RMurdock VSellis TYu J(2015)Dynamic Resource Management In a Massively Parallel Stream Processing EngineProceedings of the 24th ACM International on Conference on Information and Knowledge Management10.1145/2806416.2806449(13-22)Online publication date: 17-Oct-2015
https://dl.acm.org/doi/10.1145/2806416.2806449
Show More Cited By

Index Terms

Flood: elastic streaming MapReduce
1. Software and its engineering
  1. Software notations and tools
    1. Context specific languages
      1. Domain specific languages
  2. Software organization and properties
    1. Software system structures
      1. Software architectures

Recommendations

Synergy: sharing-aware component composition for distributed stream processing systems
Middleware'06: Proceedings of the 7th ACM/IFIP/USENIX international conference on Middleware

Many emerging on-line data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real-time. Distributed stream processing systems allow in-network stream processing to ...
Synergy: sharing-aware component composition for distributed stream processing systems
Middleware '06: Proceedings of the ACM/IFIP/USENIX 2006 International Conference on Middleware

Many emerging on-line data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real-time. Distributed stream processing systems allow in-network stream processing to ...
QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

Many emerging online data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real time. Distributed stream processing systems allow in-network stream processing to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

DEBS '10: Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems

July 2010

303 pages

ISBN:9781605589275

DOI:10.1145/1827418

Co-chair:
Peter Pietzuch
Imperial College London, UK
,
Conference Chair:
Jean Bacon
University of Cambridge, UK
,
Program Chairs:
Joe Sventek
University of Glasgow, UK
,
Ugur Cetintemel
Brown University, Rhode Island

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Demonstration

Funding Sources

Fundação para a Ciência e a Tecnologia

Conference

DEBS '10

Sponsor:

DEBS '10: The 4th ACM International Conference on Distributed Event-based Systems

July 12 - 15, 2010

Cambridge, United Kingdom

Acceptance Rates

Overall Acceptance Rate 145 of 583 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
508
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Stella GLoguinov D(2022)On High-Latency Bowtie Data Streaming2022 IEEE International Conference on Big Data (Big Data)10.1109/BigData55660.2022.10020231(75-84)Online publication date: 17-Dec-2022
https://doi.org/10.1109/BigData55660.2022.10020231
Madsen KZhou YSu LGal AWeidlich MKalogeraki VVenkasubramanian N(2016)EnormProceedings of the 10th ACM International Conference on Distributed and Event-based Systems10.1145/2933267.2933315(37-48)Online publication date: 13-Jun-2016
https://dl.acm.org/doi/10.1145/2933267.2933315
Madsen KZhou YBailey JMoffat AAggarwal Cde Rijke MKumar RMurdock VSellis TYu J(2015)Dynamic Resource Management In a Massively Parallel Stream Processing EngineProceedings of the 24th ACM International on Conference on Information and Knowledge Management10.1145/2806416.2806449(13-22)Online publication date: 17-Oct-2015
https://dl.acm.org/doi/10.1145/2806416.2806449
Xu GYu WChen ZZhang HMoulema PFu XLu C(2015)A cloud computing based system for cyber security managementInternational Journal of Parallel, Emergent and Distributed Systems10.1080/17445760.2014.92511030:1(29-45)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.1080/17445760.2014.925110
Madsen KThyssen PZhou Y(2014)Integrating fault-tolerance and elasticity in a distributed data stream processing systemProceedings of the 26th International Conference on Scientific and Statistical Database Management10.1145/2618243.2618288(1-4)Online publication date: 30-Jun-2014
https://dl.acm.org/doi/10.1145/2618243.2618288
Gedik BSchneider SHirzel MWu K(2014)Elastic Scaling for Data Stream ProcessingIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2013.29525:6(1447-1463)Online publication date: 1-Jun-2014
https://dl.acm.org/doi/10.1109/TPDS.2013.295
Tudoran RKeahey KRiteau PPanitkin SAntoniu GReed DSun XFoster I(2014)Evaluating streaming strategies for event processing across infrastructure cloudsProceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2014.89(151-159)Online publication date: 26-May-2014
https://dl.acm.org/doi/10.1109/CCGrid.2014.89
Bellavista PCorradi AReale A(2013)The QUASIT Model and Framework for Scalable Data Stream Processing with Quality of ServiceMobile Wireless Middleware, Operating Systems, and Applications10.1007/978-3-642-36660-4_7(92-107)Online publication date: 2013
https://doi.org/10.1007/978-3-642-36660-4_7
Bellavista PCorradi AReale A(2012)Design and Implementation of a Scalable and QoS-aware Stream Processing FrameworkProceedings of the 2012 IEEE International Conference on Green Computing and Communications10.1109/GreenCom.2012.54(458-467)Online publication date: 20-Nov-2012
https://dl.acm.org/doi/10.1109/GreenCom.2012.54
Heinze TEyers D(2011)Elastic complex event processingProceedings of the 8th Middleware Doctoral Symposium10.1145/2093190.2093194(1-6)Online publication date: 12-Dec-2011
https://dl.acm.org/doi/10.1145/2093190.2093194

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Synergy: sharing-aware component composition for distributed stream processing systems

Synergy: sharing-aware component composition for distributed stream processing systems

QoS-Aware Shared Component Composition for Distributed Stream Processing Systems