Impact of MapReduce Policies on Job Completion Reliability and Job Energy Consumption

Jia Chun Lin, Fang Yie Leu, Ying-ping Chen

Research output: Contribution to journalArticlepeer-review

9 Scopus citations

Abstract

Recently, MapReduce has been widely employed by many companies/organizations to tackle data-intensive problems over a large-scale MapReduce cluster. To solve machine/node failure which is inevitable in a MapReduce cluster, MapReduce employs several policies, such as input-data replication and intermediate-data replication policies. To speed up job execution, MapReduce allows reduce tasks to early fetch their required intermediate data. However, the impact of these policy combinations on the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster was not clear, where JCR is the reliability with which a MapReduce job can be completed by the cluster, whereas JEC is the energy consumed by the cluster to complete the job. Therefore, in this study, we analyze the JCR and JEC of a MapReduce cluster on four policy combinations (POCs for short) derived from two typical intermediate-data replication policies and two typical reduce-task assignment policies. The four POCs are further compared in extensive scenarios, which not only consider jobs at different scales with various parameters, but also give a MapReduce cluster two extreme parallel execution capabilities and diverse bandwidths. The analytical results enable MapReduce managers to comprehend how these POCs impact the JCR and JEC of a cluster and then select an appropriate POC based on the characteristics of their own MapReduce jobs and clusters.

Original languageEnglish
Article number6966761
Pages (from-to)1364-1378
Number of pages15
JournalIEEE Transactions on Parallel and Distributed Systems
Volume26
Issue number5
DOIs
StatePublished - 1 May 2015

Keywords

  • MapReduce
  • intermediate-data replication
  • job completion reliability
  • job energy consumption

Fingerprint Dive into the research topics of 'Impact of MapReduce Policies on Job Completion Reliability and Job Energy Consumption'. Together they form a unique fingerprint.

Cite this