LoRA explained: How a tiny add-on retrains a huge AI model

LoRA, short for low-rank adaptation, is a way to teach an already trained AI model a new skill without changing the model itself. Instead of retraining billions of numbers, it freezes them and trains a very small add-on that nudges the model’s behavior. The result is far cheaper to train and tiny to store.

A recipe book with sticky notes instead of a rewrite

Imagine a thick cookbook that a chef knows by heart. You want the chef to cook for a vegetarian restaurant. You could reprint the whole book with every recipe rewritten, or you could add a few sticky notes: swap this ingredient, skip that step. The chef still has the original book, and the notes steer the result.

LoRA works like the sticky notes. The big pretrained model stays untouched. A small set of extra numbers learns the adjustments for the new task, and you can peel them off again at any time.

Two thin matrices standing in for one giant one

A neural network stores most of what it knows in large grids of numbers called weight matrices. Ordinary fine-tuning changes every number in those grids. That needs a lot of memory, because training also has to track extra values for each trainable number.

LoRA’s idea is that the change needed for a new task can be described far more compactly. Rather than learning a full grid of updates, it learns two thin matrices. Multiplied together, they produce a full-sized correction. The size of the thin dimension is called the rank, and a small rank means very few new numbers.

In the original design, the model’s output for a layer is the frozen weights applied to the input, plus the correction from the two thin matrices. One of them starts at zero, so training begins exactly at the original model’s behavior. A scaling factor controls how strongly the correction counts.

The adapters are usually attached to the layers inside the transformer, the dominant architecture of modern language models. Which layers get adapters is a design choice, and it matters for quality (more on that below).

From GPT-3 headaches to a standard tool

LoRA was introduced in 2021 by Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen in the paper “LoRA: Low-Rank Adaptation of Large Language Models”. The authors pointed to GPT-3 175B as the problem case: keeping a fully fine-tuned copy for every task was becoming impractical.

Their reported results, which are the authors’ own claims:

• Compared with fine-tuning GPT-3 175B with the Adam optimizer, LoRA cut trainable parameters by 10,000 times and GPU memory needs by 3 times.
• Quality was on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3.
• Unlike earlier adapter methods, it adds no extra delay at inference time, because the correction can be merged into the original weights.

The paper also offers an explanation: when a model is adapted to a task, the change appears to have low “intrinsic rank”, meaning it can be captured by a small number of directions.

In 2023, Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer published QLoRA. It keeps the frozen base model in 4-bit precision and trains LoRA adapters on top. They reported fine-tuning a 65-billion-parameter model on a single 48 GB GPU while preserving full 16-bit fine-tuning quality. That made fine-tuning big open models possible on a single workstation-class card.

Where it shows up today, including Nvidia’s video model

Adapters are now a common way to specialize open models, from chat styles to company-specific vocabularies. Because an adapter is small, one base model can serve many customers or tasks by swapping adapters.

A recent example is Nvidia’s video search release, covered in our report on InternVideo2-CLIP-anomaly-detection. According to Nvidia, the model is a fine-tune of OpenGVLab’s InternVideo2-CLIP-L14 trained with vision LoRA on the last 12 vision blocks. Of roughly 373 million parameters, 589,824 are LoRA parameters, about 0.16 percent by our arithmetic.

The release also shows the two storage forms. Nvidia’s repository holds merged weights in model.safetensors, where the correction is already folded in, and the TAO checkpoint with unmerged adapters in model.pth. Merged weights run with no extra cost. Unmerged adapters can be inspected, swapped or retrained.

Cheaper does not always mean equal

LoRA is not a free replacement for full fine-tuning. In September 2025, John Schulman and colleagues at Thinking Machines Lab published an analysis, “LoRA Without Regret”. Their findings, in brief:

• LoRA matched full fine-tuning on small-to-medium instruction and reasoning datasets, but only when applied to all layers. Attention-only adapters fell behind even with matched parameter counts.
• When a dataset holds more information than the adapter can store, LoRA learns less efficiently than full fine-tuning.
• Reinforcement learning, which absorbs little information per example, matched full fine-tuning even at rank 1.
• LoRA tolerated very large batch sizes less well, and a higher rank did not close that gap.
• The best learning rate for LoRA was about 10 times the full fine-tuning rate, and roughly independent of rank.

In practice, this means a tiny adapter suits adding a style, format or narrow skill, while teaching a model large amounts of new knowledge may need more rank or full fine-tuning. QLoRA’s savings also come with some cost in training speed, since the 4-bit weights must be converted during computation.

What to watch as adapters spread

Three things are worth following. First, evidence on where the capacity limit sits, as more labs publish comparisons like Thinking Machines Lab’s. Second, adapter serving: running many small adapters on one shared base model is cheaper than hosting many full copies, and tooling for it keeps improving. Third, use beyond text. Nvidia’s video model shows LoRA applied to vision encoders, where small domain-specific gains can be added to a general model.

One caution applies to any adapter release. A small file can still change a model’s behavior substantially, so the license and testing of the adapter matter as much as those of the base model. Nvidia’s video model, for instance, is released for non-commercial research only.