A prototype simulator for the simulation of complicated hadoop framework behaviors

Distributed computing and parallel computing have become the most effective tools for solving complex problems in amount of academia and industrial fields. Among a number of distributed and parallel computing technologies, MapReduce has been proved as one of the most efficient and reliable computing model to deal with distributed tasks. Especially, Hadoop framework, a Java based open source implementation of MapReduce has been widely used and studied. Hadoop supplies tremendously abilities to process large-scale tasks with large volume of data, which significantly facilitates the big data applications. However, Hadoop is lack of a function that to predict the job processing time before actually running a job. And also. Hadoop cannot tell that how the system performs under the condition that the system parameters have been tuned. Therefore, this paper presents a prototype Hadoop simulator which aims at simulating the parameters and interactions of the Hadoop framework using agent based simulation. The experimental results show that the simulator offers potentials to be an instrument to study complicated behaviors of the Hadoop framework.