Tackling Feature Skew in Heterogeneous Federated Learning with Semantic Enhancement

A critical challenge in federated learning is data heterogeneity, compounded by varying local data distribution, thereby significantly impacting the performance of both local and global models. Prior works struggle to effectively address data heterogeneity but ignore the feature skew. In this paper, we propose a novel method, Federated Learning with Semantic Enhancement (FedSE) to address this challenge. Specifically, we introduce semantic regularization terms and adaptive local aggregation to mitigate the drawback of local knowledge insufficiency and enhance the semantics of representations. Then, based on the neural collapse theory, we initialize a simplex equiangular tight frame structure (ETF) of the classifier and maintain its stability during local training, to achieve alignment in the feature space across different clients. Extensive experiments demonstrate our method achieves SOTA performance on several standard benchmark datasets, effectively alleviating the feature skew.