Sparse autoencoders: How researchers split a model’s activations into readable features

A sparse autoencoder (SAE) is a small helper network that translates what happens inside a language model into a long list of features that people can read. For each piece of text, only a handful of those features switch on, and each one tends to match a recognizable idea. Researchers use SAEs to look inside models that are otherwise black boxes.

Why a single neuron rarely means one thing

Imagine a library with 1,000 shelves but a million books. To fit them all, the librarian puts unrelated books on the same shelf, so no shelf has a clear topic. Language models face a similar squeeze. They need to track far more concepts than they have neurons (the small computing units in each layer).

The result is that one neuron often fires for several unrelated things. Researchers call this polysemanticity. The proposed cause is superposition: the model packs many features into overlapping patterns across many neurons. Reading single neurons, then, tells you little.

Rewriting activations as a short list of active features

An SAE undoes the packing. It takes a layer’s activations, the numbers the model computes for one token, and does two things. An encoder converts them into a much longer list of numbers called latents. A decoder then rebuilds the original activations from those latents.

The key is the word sparse. Only a few latents may be active at once, so each token is explained by a small set. During training, the SAE is rewarded for rebuilding the activations accurately while using as few latents as possible. Under that pressure, each latent tends to settle on one consistent pattern, which researchers call a feature.

The decoder is effectively a dictionary. Each row is a direction in the model’s activation space, and any activation is approximated as a sum of a few dictionary entries.

A concrete case: One million features, 100 at a time

Our report on ByteDance’s Seed team shows the scale involved. According to the model card, the team released three SAEs for Qwen3-30B-A3B, each with 1,048,576 latents (2²⁰). They read the output of the model’s mixture-of-experts blocks, where the hidden state is 2,048 numbers wide.

Each SAE uses a TopK rule with k set to 100. For every token, the 100 strongest latents stay on and the rest are set to zero. That is the sparsity: about 100 out of more than a million. Each checkpoint is about 17.2 GB, according to the card.

Where the idea came from

Sparse coding is an older idea, and k-sparse autoencoders were described by Makhzani and Frey in 2013. Applying them to language models took off in 2023. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben and Lee Sharkey reported that SAEs find features that are more interpretable and monosemantic, meaning one meaning each, than alternative methods. They measured this with automated interpretability tests, and argued that the approach resolves superposition in an unsupervised and scalable way.

In 2024, Anthropic scaled the technique to a production model. It extracted millions of features from the middle layer of Claude 3 Sonnet, covering cities, scientific fields and abstract ideas such as bugs in code. One feature responded to the Golden Gate Bridge in many languages and in images. When researchers amplified it, the model began describing itself as the bridge.

Also in 2024, Leo Gao and colleagues at OpenAI introduced TopK SAEs at scale. The approach controls sparsity directly, which simplifies tuning. They trained a 16 million latent SAE on GPT-4 activations over 40 billion tokens, reported few dead latents (features that never fire), and released code and a visualizer.

What researchers do with the features

Once an SAE is trained, a feature can be studied in three ways. Researchers can look at the text that activates it most strongly and name it. They can switch it up or down to see how outputs change, which is called steering. And they can trace which features appear before a particular answer.

In Anthropic’s tests, boosting a scam-email feature led the model to draft a scam email it would normally refuse to write, and boosting a sycophancy feature made replies fawning. Anthropic noted that this shows what a model can do, not what it will do, and that it added no new capabilities.

The Seed release applies the same lens to mixture-of-experts models. Its title question is whether individual experts specialize in human-readable ways. The model card lists the paper link as “TODO”, so that claim cannot yet be checked.

Limits worth knowing before trusting a feature list

SAEs are useful, but they are not a full explanation of a model.

• Imperfect reconstruction. The rebuilt activations never match the originals exactly. ByteDance reports a fraction of variance unexplained of 12.6% at layer 4, 25.1% at layer 30 and 7.6% at layer 46. At layer 30, a quarter of the signal is not captured.
• Coverage. Anthropic stated that its features are only a small subset of what the model learned, and that mapping all of them would be prohibitively expensive.
• Representation is not mechanism. Knowing that a feature exists does not show how the model uses it. The circuits remain unknown, in Anthropic’s words.
• Cost. A million-latent SAE adds about 17 GB of weights on top of the model, per the Seed card, and the full-width activations take 4 MiB per token.
• Naming is subjective. A feature label is a human reading of examples, so two researchers may describe the same feature differently.

What to watch next

Three questions stand out. First, whether lower reconstruction error can be reached without more latents, since the middle layer of Qwen3 is the hardest case. Second, whether features found in one model transfer to another. Third, whether feature steering can prove useful for safety, which Anthropic described as something researchers still need to show.

Open releases such as ByteDance’s matter here because they let outside groups test claims on a model anyone can download. Independent checks of the unpublished paper will show whether expert specialization holds up.