More Web Proxy on the site http://driver.im/

Article

Synergy: sharing-aware component composition for distributed stream processing systems

Authors:

Thomas Repantis,

Vana KalogerakiAuthors Info & Claims

Middleware'06: Proceedings of the 7th ACM/IFIP/USENIX international conference on Middleware

Pages 322 - 341

https://doi.org/10.1007/11925071_17

Published: 27 November 2006 Publication History

Abstract

Many emerging on-line data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real-time. Distributed stream processing systems allow in-network stream processing to achieve better scalability and quality-of-service (QoS) provision. In this paper we present Synergy, a distributed stream processing middleware that provides sharing-aware component composition. Synergy enables efficient reuse of both data streams and processing components, while composing distributed stream processing applications with QoS demands. Synergy provides a set of fully distributed algorithms to discover and evaluate the reusability of available data streams and processing components when instantiating new stream applications. For QoS provision, Synergy performs QoS impact projection to examine whether the shared processing can cause QoS violations on currently running applications. We have implemented a prototype of the Synergy middleware and evaluated its performance on both PlanetLab and simulation testbeds. The experimental results show that Synergy can achieve much better resource utilization and QoS provision than previously proposed schemes, by judiciously sharing streams and processing components during application composition.

References

[1]

Chandrasekaran, S., et al.: TelegraphCQ: Continuous dataflow processing for an uncertain world. In: Proceedings of CIDR, Asilomar, CA. (2003).

Digital Library

[2]

Arasu, A., Babcock, B., Babu, S., Cieslewicz, J., Datar, M., Ito, K., Motwani, R., Srivastava, U., Widom, J.: STREAM: The Stanford data stream management system. (to appear) (2005).

[3]

Golab, L., Ozsu, M.: Update-pattern-aware modeling and processing of continuous queries. In: Proceedings of 24th ACM SIGMOD Conference, Baltimore, MD, USA. (2005).

Digital Library

[4]

Abadi, D., et al.: The design of the borealis stream processing engine. In: Proceedings of CIDR, Asilomar, CA. (2005).

[5]

Chen, L., Reddy, K., Agrawal, G.: GATES: A grid-based middleware for distributed processing of data streams. In: Proceedings of IEEE HPDC-13, Honolulu, HI. (2004).

Digital Library

[6]

Gu, X., Yu, P., Nahrstedt, K.: Optimal component composition for scalable stream processing. In: 25th IEEE ICDCS, Columbus, OH. (2005).

Digital Library

[7]

Kumar, V., Cooper, B., Cai, Z., Eisenhauer, G., Schwan, K.: Resource-aware distributed stream management using dynamic overlays. In: 25th IEEE ICDCS, Columbus, OH. (2005).

Digital Library

[8]

Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.: Networkaware operator placement for stream-processing systems. In: Proc. of 22nd ICDE. (2006).

Digital Library

[9]

Jain, N., Amini, L., Andrade, H., King, R., Park, Y., Selo, P., Venkatramani, C.: Design, implementation, and evaluation of the linear road benchmark on the stream processing core. In: Proceedings of 25th ACM SIGMOD Conference, Chicago, IL, USA. (2006).

Digital Library

[10]

PlanetLab Consortium: http://www.planet-lab.org/ (2004).

[11]

Arabshian, K., Schulzrinne, H.: An ontology-based hierarchical peer-to-peer global servicediscovery system. Journal of Ubiquitous Computing and Intelligence (JUCI) (2005).

[12]

Abdelzaher, T.: An automated profiling subsystem for QoS-aware services. In: Proc. 6th IEEE RTAS, Real-Time Technology and Applications Symposium, Washington, DC. (2000).

Digital Library

[13]

Gu, X., Nahrstedt, K., Yu, B.: SpiderNet: An integrated peer-to-peer service composition framework. In: Proceedings of IEEE HPDC-13, Honolulu, HI. (2004).

Digital Library

[14]

Rowstron, A., Druschel, P.: Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In: Proceedings of IFIP/ACM International Conference on Distributed Systems Platforms, Heidelberg, Germany. (2001).

Digital Library

[15]

Chen, F., Repantis, T., Kalogeraki, V.: Coordinated media streaming and transcoding in peer-to-peer systems. In: Proceedings of 19th IPDPS, Denver, CO. (2005).

Digital Library

[16]

Hu, N., Steenkiste, P.: Exploiting internet route sharing for large scale available bandwidth estimation. In: Proc. of Internet Measurement Conference, IMC, New Orleans, LA. (2005).

Digital Library

[17]

Tang, C., McKinley, P.: A distributed approach to topology-aware overlay path monitoring. In: Proceedings of 24th IEEE ICDCS, Tokyo, Japan. (2004).

Digital Library

[18]

Seshadri, S., Kumar, V., Cooper, B.: Optimizing multiple queries in distributed data stream systems. In: 2nd Int. IEEE Workshop on Networking Meets Databases, NetDB. (2006).

Digital Library

[19]

Cherkasova, L., Gupta, M.: Analysis of enterprise media server workloads: Access patterns, locality, content evolution, and rates of change. IEEE/ACM Transactions on Networking, TON 12(5) (2004) 781-794.

Digital Library

[20]

Zegura, E., Calvert, K., Bhattacharjee, S.: How to model an internetwork. In: Proceedings of IEEE INFOCOM, San Francisco, CA, USA. (1996).

Digital Library

[21]

Gedik, B., Liu, L.: PeerCQ: A decentralized and self-configuring peer-to-peer information monitoring system. In: Proceedings of 23rd IEEE ICDCS, Providence, RI, USA. (2003).

Digital Library

[22]

Xing, Y., Zdonik, S., Hwang, J.: Dynamic load distribution in the borealis stream processor. In: Proc. of 21st International Conference on Data Engineering, ICDE, Tokyo, Japan. (2005).

Digital Library

[23]

Kon, F., Campbell, R., Nahrstedt, K.: Using dynamic configuration to manage a scalable multimedia distributed system. Computer Communications Journal 24 (2001) 105-123.

Digital Library

[24]

Cai, W., Coulson, G., Grace, P., Blair, G., L.Mathy, Yeung, W.: The gridkit distributed resource management framework. In: Proc. of European Grid Conference, EGC. (2005).

Digital Library

[25]

Oppenheimer, D., Albrecht, J., Patterson, D., Vahdat, A.: Design and implementation tradeoffs for wide-area resource discovery. In: Proceedings of 14th IEEE HPDC-14. (2005).

Digital Library

[26]

Tai, S., Khalaf, R., Mikalsen, T.: Composition of coordinated web services. In: Proceedings of ACM/IFIP/USENIX 5th International Middleware Conference, Toronto, Canada. (2004).

Digital Library

[27]

Bartoli, A., Jimenez-Peris, R., Kemme, B., Pautasso, C., Patarin, S., Wheater, S., Woodman, S.: The adapt framework for adaptable and composable web services. IEEE Distributed Systems On Line, Web Systems Section (2005).

[28]

Amza, C., Cox, A., Zwaenepoel, W.: A comparative evaluation of transparent scaling techniques for dynamic content servers. In: Proceedings of 21st ICDE, Tokyo, Japan. (2005).

Digital Library

[29]

Colajanni, M., Grieco, R., Malandrino, D., Mazzoni, F., Scarano, V.: A scalable framework for the support of advanced edge services. In: Proc. of HPCC-05, Sorrento, Italy. (2005).

Digital Library

[30]

Karbhari, P., Rabinovich, M., Xiao, Z., Douglis, F.: ACDN: A content delivery network for applications. In: Proceedings of 21st ACM SIGMOD Conference, Madison, WI. (2002).

Digital Library

Cited By

Liu XBuyya R(2020)Resource Management and Scheduling in Distributed Stream Processing SystemsACM Computing Surveys10.1145/335539953:3(1-41)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3355399
Yang JZhang YWang JXing C(2019)Distributed Query Engine for Multiple-Query Optimization over Data StreamDatabase Systems for Advanced Applications10.1007/978-3-030-18590-9_79(523-527)Online publication date: 22-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-18590-9_79
Loyall JGillen MPaulos ABunch LCarvalho MEdmondson JSchmidt DMartignoni III ASinclair A(2011)J. P. LOYALL ET AL.Software—Practice & Experience10.1002/spe.110141:12(1459-1489)Online publication date: 1-Nov-2011
https://dl.acm.org/doi/10.1002/spe.1101
Show More Cited By

Synergy: sharing-aware component composition for distributed stream processing systems
1. Computer systems organization
  1. Architectures
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures

Recommendations

Synergy: sharing-aware component composition for distributed stream processing systems
Middleware '06: Proceedings of the ACM/IFIP/USENIX 2006 International Conference on Middleware

Many emerging on-line data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real-time. Distributed stream processing systems allow in-network stream processing to ...
QoS-Aware Shared Component Composition for Distributed Stream Processing Systems

Many emerging online data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real time. Distributed stream processing systems allow in-network stream processing to ...
Retrospective on Aurora

This experience paper summarizes the key lessons we learned throughout the design and implementation of the Aurora stream-processing engine. For the past 2 years, we have built five stream-based applications using Aurora. We first describe in detail ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

Middleware'06: Proceedings of the 7th ACM/IFIP/USENIX international conference on Middleware

November 2006

424 pages

ISBN:354049023X

Editors:
Maarten Steen
Department of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands
,
Michi Henning
Department of Computer Science, Triodia Technologies Pty Ltd, 150 Sapphire St, Holland Park 4121, Brisbane, Australia

Sponsors

ACM: Association for Computing Machinery
USENIX Assoc: USENIX Assoc
The Melbourne Convention and Visitors Bureau
IFIP

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 27 November 2006

Check for updates

Author Tags

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 203 of 948 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu XBuyya R(2020)Resource Management and Scheduling in Distributed Stream Processing SystemsACM Computing Surveys10.1145/335539953:3(1-41)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3355399
Yang JZhang YWang JXing C(2019)Distributed Query Engine for Multiple-Query Optimization over Data StreamDatabase Systems for Advanced Applications10.1007/978-3-030-18590-9_79(523-527)Online publication date: 22-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-18590-9_79
Loyall JGillen MPaulos ABunch LCarvalho MEdmondson JSchmidt DMartignoni III ASinclair A(2011)J. P. LOYALL ET AL.Software—Practice & Experience10.1002/spe.110141:12(1459-1489)Online publication date: 1-Nov-2011
https://dl.acm.org/doi/10.1002/spe.1101
Chen JRamaswamy LLowenthal DGokhale ASchmidt D(2009)Towards efficient event aggregation in a decentralized publish-subscribe systemProceedings of the Third ACM International Conference on Distributed Event-Based Systems10.1145/1619258.1619283(1-11)Online publication date: 6-Jul-2009
https://dl.acm.org/doi/10.1145/1619258.1619283
Lakshmanan GStrom RJoosen WBerbers Y(2008)Biologically-inspired distributed middleware management for stream processing systemsProceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware10.5555/1496950.1496965(223-242)Online publication date: 2-Dec-2008
https://dl.acm.org/doi/10.5555/1496950.1496965
Repantis TKalogeraki VBaldoni R(2008)Replica placement for high availability in distributed stream processing systemsProceedings of the second international conference on Distributed event-based systems10.1145/1385989.1386012(181-192)Online publication date: 1-Jul-2008
https://dl.acm.org/doi/10.1145/1385989.1386012
Branson MDouglis FFawcett BLiu ZRiabov AYe F(2007)CLASPProceedings of the 8th ACM/IFIP/USENIX international conference on Middleware10.5555/1785080.1785104(348-367)Online publication date: 26-Nov-2007
https://dl.acm.org/doi/10.5555/1785080.1785104
Branson MDouglis FFawcett BLiu ZRiabov AYe FVenkatasubramanian N(2007)CLASPProceedings of the ACM/IFIP/USENIX 2007 International Conference on Middleware10.5555/1516124.1516148(348-367)Online publication date: 26-Nov-2007
https://dl.acm.org/doi/10.5555/1516124.1516148

View Options

View options

Media

Figures

Other

Tables

View Table of Contents