Jiahao Wang

Mixture-of-Subspaces in Low-Rank Adaptation

2024 · arXiv (Cornell University)

In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language, multimodal, and diffusion models. Initially, we equivalently decompose the weights of …