Hugging Face compares every LoRA alternative: which fine-tuning method wins for LLMs

LoRA has become the de facto standard for fine-tuning large language models: it is cheap, fast, and works almost everywhere without surprises. Hugging Face decided to ask an honest question: can we do better, and if so, when?

Why LoRA holds its ground

LoRA (Low-Rank Adaptation) works simply: instead of updating all of a model's billions of weights, the method adds a pair of small low-rank matrices to key layers. The number of trainable parameters drops by a factor of 10 to 1000. That makes fine-tuning feasible even on consumer GPUs.

That is why LoRA is everywhere: it is used to further train Llama and Mistral, to create custom styles in Stable Diffusion, and to adapt enterprise LLMs to a specific domain. The Hugging Face PEFT library sees hundreds of thousands of downloads a week.

But LoRA has weak spots. At high matrix ranks (rank=64 and above), training becomes unstable. On tasks that demand precise transfer of factual knowledge, the method sometimes falls behind full fine-tuning. And under tight memory limits, for example when training on a single budget graphics card, even LoRA can be too hungry.

What Hugging Face tested

The team took the PEFT library and ran a systematic comparison of LoRA against five alternatives on real downstream tasks:

• DoRA: decomposes weights into direction and magnitude and updates them independently, bringing behavior closer to full fine-tuning
• LoRA+: a simple idea: matrices A and B are trained with different learning rates, and B gets a higher lr to speed up convergence
• rsLoRA: a normalization factor that stabilizes gradients at large rank values
• VeRA: random frozen matrices, with only tiny scaling vectors trained; it uses tens of times fewer parameters than LoRA
• GaLore: projects the gradients themselves into a low-rank space, saving optimizer memory without changing the weight architecture

Metrics: quality on held-out tasks (NLU, instruction following, summarization), peak GPU memory use, and the speed of one training epoch.

Who is crowding the leader

There is no clear winner; each method has its own profile.

DoRA consistently delivers slightly better quality than LoRA with the same number of parameters and the same memory. The difference is most visible on instruction-following and reasoning tasks. The price is slightly longer training time because of the extra weight decomposition.

rsLoRA does not improve baseline quality, but it removes instability at high ranks. If you need rank=128 or higher, rsLoRA is practically mandatory; classic LoRA starts to "drift" there.

VeRA suits scenarios with tight limits on adapter size, for example serving thousands of per-user adapters on a server, but it is slightly worse on quality.

"LoRA is still the best default choice, but knowing the alternatives

lets you squeeze out the maximum under specific conditions," the study's authors sum up.

GaLore makes it possible to train on GPUs with little VRAM without changing the weight architecture. It suits pre-training or continued pre-training, when you need to work with all the weights but memory is desperately short. Training is noticeably slower, though.

What this means

The PEFT ecosystem is maturing: instead of one universal method for every case, a toolbox is taking shape. For product teams this means one thing: before choosing a fine-tuning method, spend an hour on a comparative benchmark on your own task rather than defaulting to LoRA out of habit. The odds that an alternative will deliver a noticeable gain in your specific scenario are higher now than ever.