Huai-Chih Wang

Optimization of Spark Data Skew in Big Data Environment

2023

In order to solve the problem of uneven data distribution in the shuffle stage of Spark distributed platform, this paper first analyzes the causes of data skew in Spark platform, and then establishes a skew …