Hot AI News

Quantization: How AI models shrink to 2 bits per weight and still work

Quantization stores a model's numbers with fewer bits, so it takes less memory and runs faster. Here is how it works, where it came from and what it costs in accuracy.

Quantization: How AI models shrink to 2 bits per weight and still work
Photo illustration: Aiholics

Quantization is a way of storing a neural network’s numbers with fewer bits. A model that kept each number in 16 bits can be rewritten with 8, 4 or even 2 bits per number. The file gets much smaller and often runs faster, at the price of some accuracy.

It is the main reason capable models now fit on phones, laptops and cameras.

Rounding a model’s numbers, like saving a photo as a smaller file

Think of a photo saved as a high-quality image and then as a compressed one. The compressed copy is far smaller. At a glance it looks the same, though fine detail is lost.

Quantization does the same to a model. A neural network is mostly a huge table of numbers called weights. In the usual 16-bit format, each weight can take one of about 65,000 distinct values. With 4 bits, only 16 values are available. With 2 bits, just 4.

A concrete example from the news: Cactus Compute’s vision model Mirage has 51.3 million parameters. At 16 bits each, that would be roughly 100 MB (our arithmetic: 51.3 million times 2 bytes). Cactus says the shipped file is about 22 MB, mainly because most weight matrices use 2-bit quantization. Our report on Mirage covers the model itself.

Advertisements

Mapping many values onto a few with a scale

A research display shows a cloud of blue and violet points mapped onto four evenly spaced levels.
Photo illustration: Aiholics

The basic recipe is simple. Take the range of values a group of weights covers, from the smallest to the largest. Divide that range into evenly spaced steps, one per available value. Then store each weight as the number of the nearest step.

Hugging Face’s documentation describes the common version, called affine quantization. A real value is recovered as x = S * (x_q - Z), where x_q is the small stored integer, S is a scale and Z is a zero-point that makes sure the value 0 is represented exactly. Values outside the chosen range are clipped to the nearest edge.

Two choices shape the result:

  • Granularity. One scale for a whole tensor is cheap but coarse. One scale per channel, or per small block of weights, is more accurate but needs a little more memory.
  • Calibration. To pick ranges for activations (the values flowing through the model while it runs), engineers pass a few hundred sample inputs through the model and record what they see. Hugging Face’s guide says around 200 examples is typically enough for static quantization.

Post-training rounding versus training with the rounding in the loop

There are two broad families.

Post-training quantization takes a finished model and converts it. It needs no retraining, so it is fast and popular for large language models. Quantization-aware training inserts simulated rounding during training, so the model learns to work around the lost precision. It usually holds accuracy better at very low bit counts, but it needs the training run.

A key step for large language models came in the 2022 paper GPTQ by Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, published at ICLR 2023. GPTQ is a one-shot post-training method that uses approximate second-order information to choose rounding that hurts the output least. The authors reported quantizing a 175-billion-parameter model in about four GPU hours, with negligible accuracy loss at 3 or 4 bits, and running it on a single GPU. They also reported reasonable accuracy at 2 bits and with ternary values.

Advertisements

Going below 4 bits: Ternary weights and mixed precision

At 2 bits, plain rounding usually damages a model badly. Two ideas help.

The first is to train for low bits from the start. In February 2024, Microsoft researchers led by Shuming Ma and Furu Wei published BitNet b1.58. Every weight there is ternary, meaning only -1, 0 or 1, which works out to about 1.58 bits. The authors claim it matches a full-precision model of the same size and training data on perplexity and end tasks while using less latency, memory and energy. The paper was marked as work in progress, and those results are the authors’ own claims.

The second is mixed precision: quantize the bulk and protect the sensitive parts. Cactus says Mirage keeps token embeddings and routing weights at 4 bits and leaves some other tensors at 16-bit floating point. The company also describes a design with convolution layers that store fewer weights to begin with. Those figures come from Cactus’s launch blog and have not been independently checked.

Where quantization is used today

Nearly every on-device model relies on it. Phone chips and neural processing units (NPUs) are built to do small-integer math quickly and with little power. Quantization also lets data-center operators serve large models on fewer GPUs, and it is behind the 4-bit and 8-bit files that open-weight communities share on Hugging Face.

It matters for apps because a smaller file means a smaller download, and local inference means a photo can be processed on the device. Running locally does not by itself guarantee nothing is sent elsewhere. That depends on the app.

The costs: Lost accuracy, odd failures and uneven speedups

Lower precision is not free, and the damage is uneven.

  • Accuracy. Loss is small at 8 and 4 bits for many models, and grows at 2 bits. Tasks needing precision, such as reading small text or long reasoning chains, can degrade before simple ones do.
  • Outliers. A few unusually large values force a wide range, which wastes steps on everything else. Much of the research is about handling them.
  • Speed and energy are not automatic. Hugging Face’s guide cites third-party benchmarks on NVIDIA RTX GPUs. There, 4-bit NF4 quantization of models under 3 billion parameters used more energy than 16-bit, by 25 to 56 percent, because converting values back costs time. Batch size had a bigger effect than precision.
  • Hardware support. A format only speeds things up if the chip has fast kernels for it. A 2-bit file may need to be expanded before use on hardware without native support.
Advertisements

What to watch next

Three things are worth following. One is how well ternary and 2-bit models hold up when trained at larger scale, since BitNet’s claims are early. Another is chip support for tiny formats, which is what turns small files into real speed. The third is independent testing: vendors usually report one device and a few benchmarks, so claims such as 74 ms on a single phone are best read as a starting point rather than a general result.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like