Analyzing job completion reliability and job energy consumption for a general MapReduce infrastructure

Jia Chun Lin, Fang Yie Leu*, Ying-ping Chen

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Scopus citations

Abstract

Recently, MapReduce has been a popular distributed programming framework, which divides a job into map tasks and reduce tasks and executes these tasks in parallel over a large-scale MapReduce cluster to speed up job execution. Generally, the cluster is a master-slave infrastructure. To prevent jobs from being interrupted due to node failure, current MapReduce implementations, such as Hadoop, adopt a task-reexecution policy on the slave side, i.e., when a slave node due to failure cannot complete a task, this task will be reassigned to another available slave for reexecution. However, on the master side by default, no redundancy scheme is provided. Since this type of infrastructure has been worldwide adopted, we call it the general MapReduce infrastructure (GMI). To achieve a more reliable and energy-efficient working environment, understanding the impact of GMI on its job completion reliability (JCR) and job energy consumption (JEC) is required. In this paper, we base on a Poisson distribution to analyze GMI's JCR from a single-job perspective. After that, we accordingly derive the corresponding JEC. Through the analytical results, MapReduce managers can comprehend how GMI behaves and how their MapReduce can be improved so as to achieve a more reliable and energy-efficient MapReduce environment.

Original languageEnglish
Pages (from-to)203-214
Number of pages12
JournalJournal of High Speed Networks
Volume19
Issue number3
DOIs
StatePublished - 27 Nov 2013

Keywords

  • Job completion reliability
  • Job energy consumption
  • MapReduce
  • Master-slave infrastructure
  • Poisson distribution
  • Single-job perspective

Fingerprint Dive into the research topics of 'Analyzing job completion reliability and job energy consumption for a general MapReduce infrastructure'. Together they form a unique fingerprint.

Cite this