More Web Proxy on the site http://driver.im/

research-article

An empirical analysis of stateful operator migration for online scheduling in distributed stream processing systems

Authors:

K. Sornalakshmi,

G. VadivuAuthors Info & Claims

Volume 98, Issue C

https://doi.org/10.1016/j.micpro.2023.104811

Published: 01 April 2023 Publication History

Abstract

Streaming data is the data from sensors as well as other real-time surveillance systems. Distributed stream processing systems are the software that manages such data. Such frameworks have to deliver outcomes on the go instantly. They are susceptible to delay and malfunction or system failures. The system must be tolerant of faults and always accessible. Many variables, such as improved network arrival rates, node failures, and so on, disrupt the system's reliability. Some operators need to be relocated online from one physical resource to another to manage or reimburse a slow or failing node. In this study, we propose a co-location based systematic migration heuristic for live operator migration between physical resources using a migration map revised with costs for each migration. The suggested method evaluates continuous operator performance patterns and makes online scheduling decisions based on the same. The decisions include migrating operators during a node failure or straggling.

References

[1]

M. Hirzel, R. Soulé, S. Schneider, B. Gedik, R. Grimm, A catalog of stream processing optimizations, ACM Comput. Surv. 46 (4) (2014) 1–34.

[2]

A. McGregor, Graph stream algorithms: a survey, SIGMOD Rec. 43 (1) (May 2014) 9–20.

[3]

V. Cardellini, V. Grassi, F.Lo Presti, M. Nardelli, Optimal operator placement for distributed stream processing applications, in: DEBS 2016 - Proc. 10th ACM Int. Conf. Distrib. Event-Based Syst, 2016, pp. 69–80.

[4]

T. Lorido-Botran, J. Miguel-Alonso, J.A. Lozano, A review of auto-scaling techniques for elastic applications in cloud environments, J. Grid Comput. 12 (4) (2014) 559–592.

[5]

S. Imai, T. Chestna, C.A. Varela, Elastic scalable cloud computing using application-level migration, in: Proc. - 2012 IEEE/ACM 5th Int. Conf. Util. Cloud Comput. UCC 2012, 2012, pp. 91–98.

[6]

K. Kanoun, N. Mastronarde, D. Atienza, M. Van Der Schaar, Online energy-efficient task-graph scheduling for multicore platforms, IEEE Trans. Comput. Des. Integr. Circuits Syst. 33 (8) (2014) 1194–1207.

[7]

H. Andrade, B. Gedik, K.L. Wu, P.S. Yu, Processing high data rate streams in System S, J. Parallel Distrib. Comput. 71 (2) (2011) 145–156.

[8]

J. Ghaderi, S. Shakkottai, R. Srikant, Scheduling storms and streams in the cloud, ACM Trans. Model. Perform. Eval. Comput. Syst. 1 (4) (2016) 439–440.

[9]

B. Gedik, S. Schneider, M. Hirzel, K.L. Wu, Elastic scaling for data stream processing, IEEE Trans. Parallel Distrib. Syst. 25 (6) (2014) 1447–1463.

Digital Library

[10]

M. Nardelli, V. Cardellini, V. Grassi, F.Lo Presti, Efficient operator placement for distributed data stream processing applications, IEEE Trans. Parallel Distrib. Syst. 30 (8) (2019) 1753–1767.

[11]

L. Aniello, R. Baldoni, L. Querzoni, Adaptive online scheduling in storm, in: DEBS 2013 - Proc. 7th ACM Int. Conf. Distrib. Event-Based Syst, 2013, pp. 207–218.

[12]

T. Heinze, Z. Jerzak, G. Hackenbroich, C. Fetzer, Latency-aware elastic scaling for distributed data stream processing systems, in: DEBS 2014 - Proc. 8th ACM Int. Conf. Distrib. Event-Based Syst, 2014, pp. 13–22.

[13]

Y. Tang, B. Gedik, Autopipelining for data stream processing, IEEE Trans. Parallel Distrib. Syst. 24 (12) (Dec. 2013) 2344–2354.

[14]

T. Buddhika, R. Stern, K. Lindburg, K. Ericson, S. Pallickara, Online scheduling and interference alleviation for low-latency, high-throughput processing of data streams, IEEE Trans. Parallel Distrib. Syst. 28 (12) (2017) 3553–3569.

[15]

T. De Matteis, G. Mencagli, Proactive elasticity and energy awareness in data stream processing, J. Syst. Softw. 127 (2017) 302–319.

[16]

N. Rapolu, S. Chakradhar, A. Grama, VAYU: accelerating stream processing applications through dynamic network-aware topology re-optimization, J. Parallel Distrib. Comput. 111 (2018) 13–23.

[17]

T.Z.J. Fu, J. Ding, R.T.B. Ma, M. Winslett, Y. Yang, Z. Zhang, DRS: dynamic Resource Scheduling for Real-Time Analytics over Fast Streams, in: Proc. - Int. Conf. Distrib. Comput. Syst, 2015-July, 2015, pp. 411–420.

[18]

S. Veith et al., "Monte-Carlo Tree Search and Reinforcement Learning for Reconfiguring Data Stream Processing on Edge Computing To cite this version : HAL Id : hal-02305472 Monte-Carlo Tree Search and Reinforcement Learning for Reconfiguring Data Stream Processing on Edge," 2019.

[19]

Apache, “Apache Storm.” [Online]. Available: https://storm.apache.org/. [Accessed: 09-Jul-2020].

[20]

Apache, “Apache Spark Streaming.” [Online]. Available: https://spark.apache.org/streaming/. [Accessed: 09-Jul-2020].

[21]

“IBM Streams.” [Online]. Available: https://ibmstreams.github.io/. [Accessed: 09-Jul-2020].

[22]

M.A. Shah, J.M. Hellerstein, S. Chandrasekaran, M.J. Franklin, Flux: an adaptive partitioning operator for continuous query systems, in: Proc. - Int. Conf. Data Eng, 2003, pp. 25–36.

Cited By

Wang MZhang PZhang GSun KZhang JJin M(2025)A resilient scheduling framework for multi-robot multi-station welding flow shop scheduling against robot failuresRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10283591:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.rcim.2024.102835

Recommendations

To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing
One of the most important issues in distributed data stream processing systems is using operator migration to handle highly variable workloads cost-efficiently and adapt to the needs at any given time on demand. Operator migration is a complex process ...
Grouping Distributed Stream Query Services by Operator Similarity and Network Locality
SERVICES '08: Proceedings of the 2008 IEEE Congress on Services - Part I

Distributed stream query services must simultaneously process a large number of complex, continuous queries with stringent performance requirements while utilizing distributed processing resources. In this paper we present the design and evaluation of a ...
Operator Migration for Distributed Complex Event Processing in Device-to-Device Based Networks
M4IoT 2016: Proceedings of the 3rd Workshop on Middleware for Context-Aware Applications in the IoT

Recent times have seen a surge in the number of context-aware systems, given the proliferation of sensors and sensor-based devices, especially with the advent of new paradigms such as the Internet of Things (IoT) and device-to-device (D2D) ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Microprocessors & Microsystems

Microprocessors & Microsystems Volume 98, Issue C

Apr 2023

484 pages

ISSN:0141-9331

Issue’s Table of Contents

Copyright © 2023.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 April 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang MZhang PZhang GSun KZhang JJin M(2025)A resilient scheduling framework for multi-robot multi-station welding flow shop scheduling against robot failuresRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10283591:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.rcim.2024.102835

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents