Hadoop triggered opt/electrical data-center orchestration architecture for reducing power consumption

In this paper, a data-center network (DCN) system that distinguishes Hadoop job types and allocates optical/electrical circuits to data flows depending on the types automatically is proposed. The proposed system calculates the predicted shuffle value (PSV) of the Hadoop job and determines which type of flow is allocated by comparing the PSV and the threshold value. The PSV can be calculated from a part of the input data based on the shuffle ratio, which is known to have a relationship to the heaviness of the shuffle phase of the Hadoop job. The threshold of PSV can be dynamically changed according to the network load condition to exploit the optical circuit efficiently. By orchestrating the DCN and the Hadoop system, the proposed system achieves the reduction of power consumption. In this study, the orchestration part of the proposed DCN system is implemented and the feasibility of switching data flows between optical and electrical circuits is verified.