A Highly-Efficient Fault Tolerance Method for a Scalable Stream Processing System

An effective fault-tolerant mechanism becomes essential to the credible stream processing system (SPS). However, the traditional introduction of fault-tolerant mechanisms has a relatively negative influence on the systemic calculation efficiency. And enhancing the system scale out is an effective way to solve this problem. This paper proposes a highly-efficient fault recovery method that is for an extensible SPS. On the one hand, based on a reasonable partition and upstream backup of its internal calculation status. The SPS can monitor the bottleneck state of the operators and then migrate the upstream backup state of the operators that is in need of scale out to the new nodes to achieve dynamic scale out of the SPS; On the other hand, when a node fault in the SPS, the system can dynamically expand a secondary node for the erroneous one, and then use the upstream backup algorithm to achieve fault recovery. In this paper, a mathematical model is built for the SPS. And based on this, the fault recovery time and computational efficiency of the rapid fault tolerant method are analyzed. Experimental results have shown that the one proposed in this paper is more effective in fault tolerance which gives the system a higher calculation efficiency by reducing the frequency of upstream backup of node state.