More Web Proxy on the site http://driver.im/

Article

Efficient dynamic operator placement in a locally distributed continuous query system

Authors:

Ji WuAuthors Info & Claims

ODBASE'06/OTM'06: Proceedings of the 2006 Confederated international conference on On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE - Volume Part I

Pages 54 - 71

https://doi.org/10.1007/11914853_5

Published: 29 October 2006 Publication History

Abstract

In a distributed processing environment, the static placement of query operators may result in unsatisfactory system performance due to unpredictable factors such as changes of servers' load, data arrival rates, etc The problem is exacerbated for continuous (and long running) monitoring queries over data streams as any suboptimal placement will affect the system for a very long time In this paper, we formalize and analyze the operator placement problem in the context of a locally distributed continuous query system We also propose a solution, that is asynchronous and local, to dynamically manage the load across the system nodes Essentially, during runtime, we migrate query operators/fragments from overloaded nodes to lightly loaded ones to achieve better performance Heuristics are also proposed to maintain good data flow locality Results of a performance study shows the effectiveness of our technique.

References

[1]

A Arasu, et al Stream: The stanford stream data manager IEEE Data Eng Bull, 26(1):19-26, 2003.

[2]

A Ayad and J F Naughton Static optimization of conjunctive queries with sliding windows over infinite streams In SIGMOD, pages 419-430, 2004.

Digital Library

[3]

Y Ahmad and U Çetintemel Networked query processing for distributed streambased applications In VLDB, pages 456-467, 2004.

Digital Library

[4]

D Carney, et al Monitoring streams - a new class of data management applications In VLDB, pages 215-226, 2002.

Digital Library

[5]

D Carney, et al Operator scheduling in a data stream manager In VLDB, pages 838-849, 2003.

Digital Library

[6]

S Chandrasekaran, et al Telegraphcq: Continuous dataflow processing for an uncertain world In CIDR, 2003.

[7]

M Cherniack, et al Scalable distributed stream processing In CIDR, 2003.

[8]

P Pietzuch et al Network-aware operator placement for stream-processing systems In ICDE, pages 49, 2006.

Digital Library

[9]

M A Shah, et al Flux: An adaptive partitioning operator for continuous query systems In ICDE, pages 25-36, 2003.

[10]

U Srivastava, et al Operator Placement for In-Network Stream Query Processing In PODS, pages 250-258, 2005.

Digital Library

[11]

F Tian and D J DeWitt Tuple routing strategies for distributed eddies In VLDB, pages 333-344, 2003.

Digital Library

[12]

S Viglas and J F Naughton Rate-based query optimization for streaming information sources In SIGMOD, pages 37-48, 2002.

Digital Library

[13]

M Willebeek-LeMair and A P Reeves Strategies for dynamic load balancing on highly parallel computers IEEE Trans Parallel Distrib Syst, 4(9):979-993, 1993.

Digital Library

[14]

Y Xing, et al Dynamic load distribution in the Borealis stream processor In ICDE, pages 791-802, 2005.

Digital Library

[15]

Y Zhou, et al Adaptive reorganization of coherency-preserving dissemination tree for streaming data In ICDE, pages 55, 2006.

Digital Library

[16]

Y Zhou, et al Dynamic load management for distributed continuous query systems Unpublished manuscript, 2005 http://www.comp.nus.edu.sg/~zhouyong/ papers/op.html.

Cited By

Tang BHan HYang QXu W(2024)Operator placement for data stream processing based on publisher/subscriber in hybrid cloud-fog-edge infrastructureCluster Computing10.1007/s10586-023-04065-z27:3(2741-2759)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10586-023-04065-z
Salvaneschi GKöhler MSokolowski DHaller PErdweg SMezini M(2019)Language-integrated privacy-aware distributed queriesProceedings of the ACM on Programming Languages10.1145/33605933:OOPSLA(1-30)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1145/3360593
Rho JAzumi TNakagawa MSato KNishio N(2017)Scheduling parallel and distributed processing for automotive data stream management systemJournal of Parallel and Distributed Computing10.5555/3143043.3143183109:C(286-300)Online publication date: 1-Nov-2017
https://dl.acm.org/doi/10.5555/3143043.3143183
Show More Cited By

Efficient dynamic operator placement in a locally distributed continuous query system
1. Information systems
  1. Data management systems
    1. Database management system engines
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory

Recommendations

Distributed stream join query processing with semijoins

This paper addresses the distributed stream processing of window-based multi-way join queries considering the semijoin as a key join operator. In distributed stream processing, data streams arriving at remote sites need to be shipped to the processing ...
Combining Joint and Semi-Join Operations for Distributed Query Processing

The application of a combination of join and semi-join operations to minimize the amount of data transmission required for distributed query processing is discussed. Specifically, two important concepts that occur with the use of join operations as ...
Operator placement for in-network stream query processing
PODS '05: Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

In sensor networks, data acquisition frequently takes place at low-capability devices. The acquired data is then transmitted through a hierarchy of nodes having progressively increasing network band-width and computational power. We consider the problem ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

ODBASE'06/OTM'06: Proceedings of the 2006 Confederated international conference on On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE - Volume Part I

October 2006

1109 pages

ISBN:3540482873

Editors:
Robert Meersman
STARLab, Vrije Universiteit Brussel (VUB), Bldg G/10, Pleinlaan 2, Brussels, Belgium
,
Zahir Tari
School of Computer Science and Information Technology, RMIT University, Bld 10.10, 376-392 Swanston Street, Melbourne, VIC, Australia

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 29 October 2006

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tang BHan HYang QXu W(2024)Operator placement for data stream processing based on publisher/subscriber in hybrid cloud-fog-edge infrastructureCluster Computing10.1007/s10586-023-04065-z27:3(2741-2759)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10586-023-04065-z
Salvaneschi GKöhler MSokolowski DHaller PErdweg SMezini M(2019)Language-integrated privacy-aware distributed queriesProceedings of the ACM on Programming Languages10.1145/33605933:OOPSLA(1-30)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1145/3360593
Rho JAzumi TNakagawa MSato KNishio N(2017)Scheduling parallel and distributed processing for automotive data stream management systemJournal of Parallel and Distributed Computing10.5555/3143043.3143183109:C(286-300)Online publication date: 1-Nov-2017
https://dl.acm.org/doi/10.5555/3143043.3143183
Chakraborty RMajumdar SAnjum ASill AZhao XFarid MPallickara SCao J(2017)Priority Based Resource Scheduling Techniques for a Resource Constrained Stream Processing SystemProceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies10.1145/3148055.3148066(21-31)Online publication date: 5-Dec-2017
https://dl.acm.org/doi/10.1145/3148055.3148066
Hidalgo NWladdimiro DRosas E(2017)Self-adaptive processing graph with operator fission for elastic stream processingJournal of Systems and Software10.1016/j.jss.2016.06.010127:C(205-216)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1016/j.jss.2016.06.010
Rho JAzumi TOyama HSato KNishio N(2016)Distributed processing for automotive data stream management system on mixed single- and multi-core processorsACM SIGBED Review10.1145/2983185.298318713:3(15-22)Online publication date: 2-Aug-2016
https://dl.acm.org/doi/10.1145/2983185.2983187
Madsen KZhou YSu LGal AWeidlich MKalogeraki VVenkasubramanian N(2016)EnormProceedings of the 10th ACM International Conference on Distributed and Event-based Systems10.1145/2933267.2933315(37-48)Online publication date: 13-Jun-2016
https://dl.acm.org/doi/10.1145/2933267.2933315
Cardellini VGrassi VLo Presti FNardelli MGal AWeidlich MKalogeraki VVenkasubramanian N(2016)Optimal operator placement for distributed stream processing applicationsProceedings of the 10th ACM International Conference on Distributed and Event-based Systems10.1145/2933267.2933312(69-80)Online publication date: 13-Jun-2016
https://dl.acm.org/doi/10.1145/2933267.2933312
Madsen KZhou YBailey JMoffat AAggarwal Cde Rijke MKumar RMurdock VSellis TYu J(2015)Dynamic Resource Management In a Massively Parallel Stream Processing EngineProceedings of the 24th ACM International on Conference on Information and Knowledge Management10.1145/2806416.2806449(13-22)Online publication date: 17-Oct-2015
https://dl.acm.org/doi/10.1145/2806416.2806449
Madsen KThyssen PZhou Y(2014)Integrating fault-tolerance and elasticity in a distributed data stream processing systemProceedings of the 26th International Conference on Scientific and Statistical Database Management10.1145/2618243.2618288(1-4)Online publication date: 30-Jun-2014
https://dl.acm.org/doi/10.1145/2618243.2618288
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents