Optimization for Speculative Execution in Heterogeneous MapReduce Environment

MapReudce proposed by Google is an open-source distributed computing framework for parallel processing of large-scale data. Speculative execution is a problem in the MapReduce. Because a job submitted by client can not finish until all map and reduce tasks finished. Sometimes, because of the heterogeneous environment (the computing capabilities of each tasktracker is different), some tasktrackers process tasks slower than others, we called them stragglers. In case of the stragglers, the tasks assigned will incur lower performance as well as entire execution time delay. The purpose of our research is to optimize the speculative execution through improving LATE [1] scheduler, we will solve the issue still exist in LATE, that the average progress score is unstable. To tackle this issue, we proposed that take process bandwidth along with the average progress rate of each phase into considered. We use two applications to simulate our proposal. One is word count and anther is sort. The result showed that our proposal can decrease the time of speculative execution from 3% to 15%.