Resource Optimization for Speculative Execution in a MapReduce Cluster
Refereed conference paper presented and published in conference proceedings


Full Text

Times Cited
Web of Science0WOS source URL (as at 17/11/2020) Click here for the latest count

Other information
AbstractThe MapReduce paradigm is now the de facto standard for large-scale data analytics. In this paper we address the resource management issues in MapReduce Cluster. Speculative execution (task backup) plays an important role in resource management. We propose two different strategies and build two models to formulate the backup issue as an optimization problem when the cluster is lightly loaded. Moreover, we present an Enhanced Speculative Execution (ESE) algorithm when the cluster is heavily loaded and adopt the approximate analysis to get an optimal value for the parameter in the algorithm. The simulation results show that the algorithm can reduce the job completion time by 50% while consuming much less resource compared to the naive method without backup.
All Author(s) ListXu HL, Lau WC
Name of Conference21st IEEE International Conference on Network Protocols (ICNP)
Start Date of Conference07/10/2013
End Date of Conference10/10/2013
Place of ConferenceGottingen
Country/Region of ConferenceGermany
Year2013
Month1
Day1
PublisherIEEE
ISBN978-1-4799-1270-4
eISBN978-1-4799-1270-4
ISSN1092-1648
LanguagesEnglish-United Kingdom
Keywordsjob scheduling; MapReduce; speculative execution; theoretical analysis
Web of Science Subject CategoriesTelecommunications

Last updated on 2020-18-11 at 00:09