Junxiao Wang

Demystify Self-Attention in Vision Transformers from a Semantic Perspective: Analysis and Application

2022 · arXiv (Cornell University)

Self-attention mechanisms, especially multi-head self-attention (MSA), have achieved great success in many fields such as computer vision and natural language processing. However, many existing vision transformer (ViT) works simply inherent transformer designs from NLP to …

Exploring Amplified Heterogeneity Arising From Heavy-Tailed Distributions in Federated Learning

2024 · IEEE Transactions on Mobile Computing

Federated Learning (FL) has emerged as a privacy-preserving paradigm enabling collaborative model training among distributed clients. However, current FL methods operate under the closed-world assumption, i.e., all local training data originates from a global labeled …