More Web Proxy on the site http://driver.im/

article

Nephele streaming: stream processing under QoS constraints at scale

Authors:

Björn Lohrmann,

Daniel Warneke,

Odej KaoAuthors Info & Claims

Cluster Computing, Volume 17, Issue 1

Pages 61 - 78

https://doi.org/10.1007/s10586-013-0281-8

Published: 01 March 2014 Publication History

Abstract

The ability to process large numbers of continuous data streams in a near-real-time fashion has become a crucial prerequisite for many scientific and industrial use cases in recent years. While the individual data streams are usually trivial to process, their aggregated data volumes easily exceed the scalability of traditional stream processing systems.

At the same time, massively-parallel data processing systems like MapReduce or Dryad currently enjoy a tremendous popularity for data-intensive applications and have proven to scale to large numbers of nodes. Many of these systems also provide streaming capabilities. However, unlike traditional stream processors, these systems have disregarded QoS requirements of prospective stream processing applications so far.

In this paper we address this gap. First, we analyze common design principles of today's parallel data processing frameworks and identify those principles that provide degrees of freedom in trading off the QoS goals latency and throughput. Second, we propose a highly distributed scheme which allows these frameworks to detect violations of user-defined QoS constraints and optimize the job execution without manual interaction. As a proof of concept, we implemented our approach for our massively-parallel data processing framework Nephele and evaluated its effectiveness through a comparison with Hadoop Online.

For an example streaming application from the multimedia domain running on a cluster of 200 nodes, our approach improves the processing latency by a factor of at least 13 while preserving high data throughput when needed.

References

[1]

Hadoop online prototype--Google project hosting (2012). http://code.google.com/p/hop/

[2]

Justin.tv--streaming live video broadcasts for everyone (2012). http://www.justin.tv/

[3]

Livestream--be there (2012). http://www.livestream.com/

[4]

Nathanmarz/storm--GitHub (2012). https://github.com/nathanmarz/storm

[5]

Stratosphere--above the clouds (2012). http://stratosphere.eu/

[6]

USTREAM, you're on (2012). http://www.ustream.tv/

[7]

Welcome to apache Hadoop! (2012). http://http://hadoop.apache.org/

[8]

Xuggle (2012). http://http://www.xuggle.com/

[9]

Abadi, D., Ahmad, Y., Balazinska, M., Cetintemel, U., Cherniack, M., Hwang, J., Lindner, W., Maskey, A., Rasin, A., Ryvkina, E., et al.: The design of the Borealis stream processing engine. In: Second Biennial Conference on Innovative Data Systems Research (CIDR '05), pp. 277---289 (2005)

[10]

Abadi, D., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.: Aurora: a new model and architecture for data stream management. VLDB J. 12(2), 120---139 (2003)

Digital Library

[11]

Aldinucci, M., Danelutto, M.: Stream parallel skeleton optimization. In: Proc. of the 11th IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS '99), pp. 955---962. IASTED/ACTA Press, Cambridge (1999). ftp://ftp.di.unipi.it/pub/Papers/aldinuc/302-114.ps.gz

[12]

Alexandrov, A., Ewen, S., Heimel, M., Hueske, F., Kao, O., Markl, V., Nijkamp, E., Warneke, D.: MapReduce and PACT--comparing data parallel programming models. In: Proc. of the 14th Conference on Database Systems for Business, Technology, and Web (BTW '11), pp. 25---44. GI, Bonn (2011)

[13]

Babu, S., Widom, J.: Continuous queries over data streams. SIGMOD Rec. 30, 109---120 (2001)

Digital Library

[14]

Battré, D., Ewen, S., Hueske, F., Kao, O., Markl, V., Warneke, D.: Nephele/PACTs: a programming model and execution framework for web-scale analytical processing. In: Proc. of the 1st ACM Symposium on Cloud Computing (SoCC '10), pp. 119---130. ACM, New York (2010)

Digital Library

[15]

Battré, D., Hovestadt, M., Lohrmann, B., Stanik, A., Warneke, D.: Detecting bottlenecks in parallel DAG-based data flow programs. In: Proc. of the 2010 IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS '10), pp. 1---10. IEEE Press, New York (2010)

[16]

Borkar, V., Carey, M., Grover, R., Onose, N., Vernica, R.: Hyracks: a flexible and extensible foundation for data-intensive computing. In: Proc. of the 2011 IEEE 27th International Conference on Data Engineering (ICDE '11), pp. 1151---1162. IEEE Press, New York (2011). http://dx.doi.org/10.1109/ICDE.2011.5767921.

[17]

Cherniack, M., Balakrishnan, H., Balazinska, M., Carney, D., Cetintemel, U., Xing, Y., Zdonik, S.: Scalable distributed stream processing. In: Proc. of the First Biennial Conference on Innovative Data Systems Research (CIDR '03), pp. 257---268 (2003)

[18]

Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M., Elmeleegy, K., Sears, R.: MapReduce online. In: Proc. of the 7th USENIX Conference on Networked Systems Design and Implementation (NSDI '10), USENIX Association, Berkeley (2010). p. 21

Digital Library

[19]

Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107---113 (2008)

Digital Library

[20]

Elnozahy, E.N.M., Alvisi, L., Wang, Y.M., Johnson, D.B.: A survey of rollback-recovery protocols in message-passing systems. ACM Comput. Surv. 34(3), 375---408 (2002).

Digital Library

[21]

Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. Oper. Syst. Rev. 41(3), 59---72 (2007)

Digital Library

[22]

Lam, W., Liu, L., Prasad, S., Rajaraman, A., Vacheri, Z., Doan, A.: Muppet: mapreduce-style processing of fast data. Proc. VLDB Endow. 5(12), 1814---1825 (2012)

Digital Library

[23]

Li, B., Mazur, E., Diao, Y., McGregor, A., Shenoy, P.: A platform for scalable one-pass analytics using mapreduce. In: Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (SIGMOD '11), pp. 985---996. ACM, New York (2011)

Digital Library

[24]

Lohrmann, B., Warneke, D., Kao, O.: Massively-parallel stream processing under QoS constraints with Nephele. In: Proceedings of the 21st International Symposium on High-Performance Parallel and Distributed Computing (HPDC '12), pp. 271---282. ACM, New York (2012)

[25]

Motwani, R., Widom, J., Arasu, A., Babcock, B., Babu, S., Datar, M., Manku, G., Olston, C., Rosenstein, J., Varma, R.: Query processing, approximation, and resource management in a data stream management system. In: First Biennial Conference on Innovative Data Systems Research (CIDR '03), pp. 245---256 (2003)

[26]

Murray, D., Schwarzkopf, M., Smowton, C., Smith, S., Madhavapeddy, A., Hand, S.: CIEL: a universal execution engine for distributed data-flow computing. In: Proc. of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI '11), USENIX Association, Berkeley (2011). p. 9

[27]

Neumeyer, L., Robbins, B., Nair, A., Kesari, A.: S4: distributed stream computing platform. In: 2010 IEEE International Conference on Data Mining Workshops (ICDMW '10), pp. 170---177. IEEE Press, New York (2010)

[28]

Warneke, D., Kao, O.: Exploiting dynamic resource allocation for efficient parallel data processing in the cloud. IEEE Trans. Parallel Distrib. Syst. 22(6), 985---997 (2011)

Digital Library

Cited By

Jamieson SForshaw MKemme BRiviere ESchiavoni VPasin M(2023)On Improving Streaming System Autoscaler Behaviour using Windowing and Weighting MethodsProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596886(68-79)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596886
Cardellini VLo Presti FNardelli MRusso G(2022)Runtime Adaptation of Data Stream Processing Systems: The State of the ArtACM Computing Surveys10.1145/351449654:11s(1-36)Online publication date: 9-Sep-2022
https://dl.acm.org/doi/10.1145/3514496
Crankshaw DSela GMo XZumar CStoica IGonzalez JTumanov AFonseca RDelimitrou COoi B(2020)InferLineProceedings of the 11th ACM Symposium on Cloud Computing10.1145/3419111.3421285(477-491)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3419111.3421285
Show More Cited By

Nephele streaming: stream processing under QoS constraints at scale
1. Software and its engineering
  1. Software organization and properties
    1. Software system structures

Recommendations

Massively-parallel stream processing under QoS constraints with Nephele
HPDC '12: Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing

Today, a growing number of commodity devices, like mobile phones or smart meters, is equipped with rich sensors and capable of producing continuous data streams. The sheer amount of these devices and the resulting overall data volumes of the streams ...
A configurable and executable model of Spark Streaming on Apache YARN

Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, ...
Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark
SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

With the ubiquity of real-time data, organizations need streaming systems that are scalable, easy to use, and easy to integrate into business applications. Structured Streaming is a new high-level streaming API in Apache Spark based on our experience ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Cluster Computing

Cluster Computing Volume 17, Issue 1

March 2014

148 pages

ISSN:1386-7857

Issue’s Table of Contents

Copyright © Copyright © 2014 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 March 2014

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jamieson SForshaw MKemme BRiviere ESchiavoni VPasin M(2023)On Improving Streaming System Autoscaler Behaviour using Windowing and Weighting MethodsProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596886(68-79)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596886
Cardellini VLo Presti FNardelli MRusso G(2022)Runtime Adaptation of Data Stream Processing Systems: The State of the ArtACM Computing Surveys10.1145/351449654:11s(1-36)Online publication date: 9-Sep-2022
https://dl.acm.org/doi/10.1145/3514496
Crankshaw DSela GMo XZumar CStoica IGonzalez JTumanov AFonseca RDelimitrou COoi B(2020)InferLineProceedings of the 11th ACM Symposium on Cloud Computing10.1145/3419111.3421285(477-491)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3419111.3421285
Liu XBuyya R(2020)Resource Management and Scheduling in Distributed Stream Processing SystemsACM Computing Surveys10.1145/335539953:3(1-41)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3355399
Röger HBhowmik SRothermel K(2019)Combining it allProceedings of the 20th International Middleware Conference10.1145/3361525.3361551(255-267)Online publication date: 9-Dec-2019
https://dl.acm.org/doi/10.1145/3361525.3361551
Röger HMayer R(2019)A Comprehensive Survey on Parallelization and Elasticity in Stream ProcessingACM Computing Surveys10.1145/330384952:2(1-37)Online publication date: 30-Apr-2019
https://dl.acm.org/doi/10.1145/3303849
Mencagli GDazzi PTonci NPierre GFerreira PShrira L(2018)SpinStreamsProceedings of the 19th International Middleware Conference10.1145/3274808.3274814(66-79)Online publication date: 26-Nov-2018
https://dl.acm.org/doi/10.1145/3274808.3274814
Liu XDastjerdi ACalheiros RQu CBuyya R(2017)A Stepwise Auto-Profiling Method for Performance Optimization of Streaming ApplicationsACM Transactions on Autonomous and Adaptive Systems10.1145/313261812:4(1-33)Online publication date: 14-Nov-2017
https://dl.acm.org/doi/10.1145/3132618
Chothia ZLiagouris JDimitrova DRoscoe T(2017)Online Reconstruction of Structural Information from Datacenter LogsProceedings of the Twelfth European Conference on Computer Systems10.1145/3064176.3064195(344-358)Online publication date: 23-Apr-2017
https://dl.acm.org/doi/10.1145/3064176.3064195
Marcu OTudoran RNicolae BCostan AAntoniu GPérez-Hernández M(2017)Exploring Shared State in Key-Value Store for Window-Based Multi-Pattern Streaming AnalyticsProceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing10.1109/CCGRID.2017.126(1044-1052)Online publication date: 14-May-2017
https://dl.acm.org/doi/10.1109/CCGRID.2017.126
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents