research-article

Potential-driven load distribution for distributed data stream processing

Authors:

Weihan Wang,

Mohamed A. Sharaf,

Shimin Guo,

M. Tamer ÖzsuAuthors Info & Claims

SSPS '08: Proceedings of the 2nd international workshop on Scalable stream processing system

Pages 13 - 22

https://doi.org/10.1145/1379272.1379277

Published: 29 March 2008 Publication History

Get Access

Abstract

A large class of applications require real-time processing of continuous stream data resulting in the development of data stream management systems (DSMS). Since many of these applications are distributed, distributed DSMSs are starting to receive attention. In this paper, we focus on an important issue in distributed DSMS operation, namely load distribution to minimize end-to-end latency. We identify the often conflicting requirements of load distribution, and propose a "potential-driven" load distribution approach to mimic the movements of objects in the physical world. Our approach also takes into account heterogeneous machines, different network conditions, and resource constraints. We present experimental results that investigate our algorithms from various aspects, and show that they outperform existing techniques in terms of end-to-end latency.

References

[1]

D. J. Abadi et al. Aurora: a new model and architecture for data stream management. The VLDB Journal, 12(2), 2003.

Digital Library

Google Scholar

[2]

D. J. Abadi et al. The design of the borealis stream processing engine. In CIDR, 2005.

Google Scholar

[3]

Y. Ahmad and U. Çetintemel. Network-aware query processing for stream-based applications. In VLDB, 2004.

Digital Library

Google Scholar

[4]

S. Chandrasekaran et al. TelegraphCQ: Continuous dataflow processing for an uncertain world. In CIDR, 2003.

Google Scholar

[5]

X. Gu, P. S. Yu, and H. Wang. Adaptive load diffusion for multiway windowed stream joins. In ICDE, 2007.

Crossref

Google Scholar

[6]

R. Motwani et al. Query processing, approximation, and resource management in a data stream management system. In CIDR, 2003.

Google Scholar

[7]

P. Pietzuch et al. Network-aware operator placement for stream-processing systems. In ICDE, 2006.

Digital Library

Google Scholar

[8]

M. A. Shah, J. M. Hellerstein, S. Chandrasekaran, and M. J. Franklin. Flux: An adaptive partitioning operator for continuous query systems. In ICDE, 2003.

Digital Library

Google Scholar

[9]

U. Srivastava, K. Munagala, and J. Widom. Operator placement for in-network stream query processing. In PODS, 2005.

Digital Library

Google Scholar

[10]

Y. Xing. Load distribution for distributed stream processing. In EDBT Ph.D. Workshop, 2004.

Digital Library

Google Scholar

[11]

Y. Xing et al. Dynamic load distribution in the borealis stream processor. In ICDE, 2005.

Digital Library

Google Scholar

[12]

Y. Xing, J.-H. Hwang, U. Çetintemel, and S. Zdonik. Providing resiliency to load variations in distributed stream processing. In VLDB, 2006.

Digital Library

Google Scholar

[13]

Y. Zhou, B. C. Ooi, and K.-L. Tan. Dynamic load management for distributed continuous query systems. In ICDE, 2005.

Digital Library

Google Scholar

[14]

Y. Zhou, B. C. Ooi, K.-L. Tan, and J. Wu. Efficient dynamic operator placement in a locally distributed continuous query system. In LNCS 4275, OTM, 2006.

Digital Library

Google Scholar

Cited By

View all

Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Jiang JZhang ZCui BTong YXu N(2017)StroMAX: Partitioning-Based Scheduler for Real-Time Stream Processing SystemDatabase Systems for Advanced Applications10.1007/978-3-319-55699-4_17(269-288)Online publication date: 22-Mar-2017
https://doi.org/10.1007/978-3-319-55699-4_17
Tinati RWang XBrown ITiropanis THall WGangemi ALeonardi SPanconesi A(2015)A Streaming Real-Time Web Observatory Architecture for Monitoring the Health of Social MachinesProceedings of the 24th International Conference on World Wide Web10.1145/2740908.2743977(1149-1154)Online publication date: 18-May-2015
https://dl.acm.org/doi/10.1145/2740908.2743977
Show More Cited By

Index Terms

Potential-driven load distribution for distributed data stream processing
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Parallel and distributed DBMSs

Recommendations

Bounding substreams in distributed stream processing
Abstract
A common problem in distributed stream processing is to split a stream into finite chunks of messages (substreams) and to determine their boundaries: stateful streaming operators should clear outdated state; time window operators ...
Highlights
- Punctuations can be inefficient for substreams bounding due to high network overhead.
Practical Storage-Compute Elasticity for Stream Data Processing
Middleware '23: Proceedings of the 24th International Middleware Conference: Industrial Track

Stream processing pipelines need to handle workload fluctuations (e.g., daily patterns, popularity spikes) by scaling up/down the resources contributed to running jobs. While there have been efforts proposing auto-scaling mechanisms for stream processing ...
Distributed processing of continuous sliding-window k-NN queries for data stream filtering

A sliding-window k-NN query (k-NN/w query) continuously monitors incoming data stream objects within a sliding window to identify k closest objects to a query. It enables effective filtering of data objects streaming in at high rates from potentially ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SSPS '08: Proceedings of the 2nd international workshop on Scalable stream processing system

March 2008

99 pages

ISBN:9781595939630

DOI:10.1145/1379272

Conference Chair:
Byung S. Lee
University of Vermont

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 March 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

EDBT '08

EDBT '08: 11th International Conference on Extending Database Technology

March 29, 2008

Nantes, France

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
279
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)6

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Volnes EPlagemann TGoebel V(2024)To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream ProcessingIEEE Communications Surveys & Tutorials10.1109/COMST.2023.333095326:1(670-705)Online publication date: Sep-2025
https://doi.org/10.1109/COMST.2023.3330953
Jiang JZhang ZCui BTong YXu N(2017)StroMAX: Partitioning-Based Scheduler for Real-Time Stream Processing SystemDatabase Systems for Advanced Applications10.1007/978-3-319-55699-4_17(269-288)Online publication date: 22-Mar-2017
https://doi.org/10.1007/978-3-319-55699-4_17
Tinati RWang XBrown ITiropanis THall WGangemi ALeonardi SPanconesi A(2015)A Streaming Real-Time Web Observatory Architecture for Monitoring the Health of Social MachinesProceedings of the 24th International Conference on World Wide Web10.1145/2740908.2743977(1149-1154)Online publication date: 18-May-2015
https://dl.acm.org/doi/10.1145/2740908.2743977
Gounaris ATsamoura EManolopoulos Y(2013)Adaptive Query Processing in Distributed SettingsAdvanced Query Processing10.1007/978-3-642-28323-9_9(211-236)Online publication date: 2013
https://doi.org/10.1007/978-3-642-28323-9_9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Bounding substreams in distributed stream processing

Practical Storage-Compute Elasticity for Stream Data Processing

Distributed processing of continuous sliding-window k-NN queries for data stream filtering