Abstract
Data streaming applications are becoming more and more common due to the rapid development in the areas such as sensor networks, multimedia streaming, and on-line data mining, etc. These applications are often running in a decentralized, distributed environment. The requirements for processing large volumes of streaming data at real time have posed many great design challenges. It is critical to optimize the ongoing resource consumption of multiple, distributed, cooperating, processing units. In this paper, we consider a generic model for the general stream data processing systems. We address the resource allocation problem for a collection of processing units so as to maximize the weighted sum of the throughput of different streams. Each processing unit may require multiple input data streams simultaneously and produce one or many valuable output streams. Data streams flow through such a system after processing at multiple processing units. Based on this framework, we develop distributed algorithms for finding the best resource allocation schemes in such data stream processing networks. Performance analysis on the optimality and complexity of these algorithms are also provided.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Baccelli, F., Liu, Z.: On the Execution of Parallel Programs on Multiprocessor Systems-A Queuing Theory Approach. Journal of the ACM 37(2), 373–417 (1990)
Baccelli, F., Liu, Z.: On the Stability Condition of a Precedence-based Queueing Discipline. Adv. Appl. Prob. 21, 883–887 (1989)
Baccelli, F., Makowski, F., Towsley, D.: Acyclic Fork-Join Queueing Networks. J. ACM 36(3), 615–642 (1989)
Baldwin, C., Clark, K.B., Magretta, J., Dyer, J.H., Fisher, M., Fites, D.V.: Harvard Business Review on Managing the Value Chain. Harvard Business School Press, Boston (2000)
Brandstdt, A., Bang Le, V., Spinrad, J.P.: Graph Classes: A Survey. SIAM, Philadelphia (1999)
Coffman, E., Lueker, G.: Probabilistic Analysis of Packing and Partitioning Algorithms. Wiley, Chichester (1991)
Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 2nd edn. MIT Press and McGraw-Hill (2001)
Mas-Collel, A., Whinston, M., Green, J.: Microeconomic Theory. Oxford University Press, Oxford (1995)
Sharp, J.A. (ed.): Data Flow Computing. Ablex Publication Corp. (1991)
Simchi-Levi, D., Kaminsky, P., Simchi-Levi, E.: Designing and Managing the Supply Chain, 2nd edn. McGraw-Hill/Irwin (2002)
Tang, K., Liu, Z., Xia, C., Zhang, L.: Distributed Resource Allocation for Stream Processing Systems. IBM Research Report (2006)
Viglas, S., Naughton, J.: Rate-Based Query Optimization for Streaming Information Sources. ACM SIGMOD (2002)
Xi, B., Liu, Z., Raghavachari, M., Xia, C., Zhang, L.: A Smart Hill-Climbing Algorithm for Application Server Configuration. In: Proceedings of the 13th International Conference on World Wide Web, pp. 287–296 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tang, A., Liu, Z., Xia, C., Zhang, L. (2006). Distributed Resource Allocation for Stream Data Processing. In: Gerndt, M., Kranzlmüller, D. (eds) High Performance Computing and Communications. HPCC 2006. Lecture Notes in Computer Science, vol 4208. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11847366_10
Download citation
DOI: https://doi.org/10.1007/11847366_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-39368-9
Online ISBN: 978-3-540-39372-6
eBook Packages: Computer ScienceComputer Science (R0)