A speculative execution strategy based on node classification and hierarchy index mechanism for heterogeneous Hadoop systems

MapReduce (MR) has been widely used to process distributed large data sets. MRV2 working on Yarn, as a more advanced programing model, has gained lots of concerns. Meanwhile, speculative execution is known as an approach for dealing with same problems by backing up those tasks running on a low performance machine to a higher one. In this paper, we have modified some pitfalls and taken heterogeneous environment into consideration. Besides, Node classification is used and a novel hierarchy index mechanism is created. We also have implemented it in Hadoop-2.6 and the strategy above is called Speculation-NC while optimized Hadoop is called Hadoop-NC. Experiment results show that our method can correctly backup a task, improve the performance of MRV2 and decrease the execution time and resource consumption compared with traditional strategies.