Cited By
View all- KAWARASAKI MWATANABE H(2015)System Status Aware Hadoop Scheduling Methods for Job Performance ImprovementIEICE Transactions on Information and Systems10.1587/transinf.2014EDP7385E98.D:7(1275-1285)Online publication date: 2015
Impact of Data Transfer to Hadoop Job Performance: Architectural Analysis and Experiment
Article No.: 63, Pages 1 - 6
Abstract
Hadoop is a distributed processing platform for analyzing a large amount of data. It uses MapReduce framework for distributed parallel processing where reduce computation uses all the relevant map computation results obtained in other nodes, thus needs massive amounts of data transfer between nodes. This paper explores how such interdependence between nodes affects the job performance in Hadoop cluster and clarifies the mechanism of job performance deterioration. For this purpose, we built two kind of experimental Hadoop clusters using real machines in our laboratory and virtual machines on Amazon EC2 and tracked the progress of tasks which proceeds in parallel. As a result, we revealed that delay in one task caused by congestions in disk I/O or data transfer propagates to other tasks and deteriorates the overall job performance significantly. Furthermore, we found that speculative task execution brings adverse effects when the task delay is caused by disk I/O.
References
[1]
Amazon EC2. http://aws.amazon.com/ec2.
[2]
Apache Hadoop. http://hadoop.apache.org.
[3]
Y. Chen, A. Ganapathi, R. Griffith, and R. Katz. The Case for Evaluating MapReduce Performance Using Workload Suites. In MASCOTS, pages 390-399, 2011.
[4]
M. Chowdhury, M. Zaharia, J. Ma, M. I. Jordan, and I. Stoika. Managing data transfers in computer clusters with orchestra. In SIGCOMM, pages 98-109, 2011.
[5]
J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, pages 137-150, 2004.
[6]
A. Konwinski. Improving MapReduce Performance in Heterogeneous Environments. Technical Report of EECS Department, University of California, Berkeley, 2009.
[7]
T. White. Hadoop: The Definitive Guide, 3rd Edition, O'Reilly Media / Yahoo Press, California, 2012.
[8]
M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In EuroSys, 2010.
[9]
M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Job Scheduling for Multi-User MapReduce Clusters. Technical Report of EECS Department, University of California, Berkeley, 2009.
Index Terms
- Impact of Data Transfer to Hadoop Job Performance: Architectural Analysis and Experiment
Recommendations
Survey on improving the performance of MapReduce in Hadoop
NISS '21: Proceedings of the 4th International Conference on Networking, Information Systems & SecurityHadoop has become the most popular and the most used platform in distributed data processing, Hadoop is also an open-source software that implements the MapReduce model for processing big data, it has taken a large part in scientific research in the ...
Performance Optimization for Short MapReduce Job Execution in Hadoop
CGC '12: Proceedings of the 2012 Second International Conference on Cloud and Green ComputingHadoop MapReduce is a widely used parallel computing framework for solving data-intensive problems. To be able to process large-scale datasets, the fundamental design of the standard Hadoop places more emphasis on high-throughput of data than on job ...
G-Hadoop: MapReduce across distributed data centers for data-intensive computing
Recently, the computational requirements for large-scale data-intensive analysis of scientific data have grown significantly. In High Energy Physics (HEP) for example, the Large Hadron Collider (LHC) produced 13 petabytes of data in 2010. This huge ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2014
265 pages
Copyright © 2014 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 March 2014
Check for updates
Author Tags
Qualifiers
- Article
Conference
ACM SE '14
Acceptance Rates
Overall Acceptance Rate 502 of 1,023 submissions, 49%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 69Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Dec 2024
Other Metrics
Citations
Cited By
View all- KAWARASAKI MWATANABE H(2015)System Status Aware Hadoop Scheduling Methods for Job Performance ImprovementIEICE Transactions on Information and Systems10.1587/transinf.2014EDP7385E98.D:7(1275-1285)Online publication date: 2015
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in