Hybrid language models: How mixing attention with cheaper layers shrinks memory

A hybrid language model is a neural network that uses more than one kind of layer to read text. Most keep a small number of attention layers, which look back over everything written so far, and fill the rest with cheaper layers that only keep a compact summary. The goal is a model that uses less memory and runs faster on long inputs.

Liquid AI’s new LFM3-1.7B, covered in our weekly roundup of new AI models, is described on its card as using a new tri-hybrid architecture.

Why attention gets expensive as the text gets longer

Picture a reader who, before writing each new word, rereads their entire notebook. That is attention. It is thorough, and it is why modern chatbots follow long conversations so well. But the notebook grows with every word, and the rereading never stops.

In a standard Transformer, each layer stores a key and a value for every earlier token. This store is called the KV cache (key-value cache). It grows with the length of the input and takes up memory, which is scarce on a phone or laptop. The work of comparing every token with every other token also grows quickly as inputs get longer.

A hybrid design asks a simple question: does every layer really need to reread the whole notebook?

Swapping most layers for ones that keep a running summary

The cheaper layers come in a few types. A short convolution looks only at the last few tokens, like glancing at the previous few words. A state space model (SSM) or recurrent layer carries a fixed-size summary forward and updates it with each new token, like a reader who keeps a single page of notes instead of the whole notebook.

Because the summary has a fixed size, memory does not grow with input length. Because only the few attention layers keep a KV cache, the cache shrinks in proportion to how many attention layers remain.

The attention layers that stay act as a safety net. They can still pull an exact detail from far back in the text, such as a name mentioned 20 pages earlier. The cheaper layers handle the routine work of tracking local context.

From Transformers to Mamba to mixed stacks

The Transformer, introduced in 2017 by Vaswani and colleagues in the paper “Attention Is All You Need,” dropped recurrent and convolutional layers entirely and relied on attention alone. That made training fast and results strong, and it became the default design.

In December 2023, Albert Gu and Tri Dao published Mamba. They argued that earlier cheap alternatives, including linear attention, gated convolutions and structured state space models, trailed attention because they could not do content-based reasoning. Their fix was a selective state space model, in which the layer’s parameters depend on the input, so the model can decide token by token what to keep or discard. They reported about 5x higher inference throughput than Transformers and costs that scale linearly with sequence length. Those are the authors’ own claims.

In 2024, a team around AI21 Labs published Jamba, which interleaves Transformer and Mamba layers and adds mixture-of-experts layers in some places. The paper reports results for context up to 256K tokens, a configuration that fits on a single 80GB GPU, and a smaller memory footprint than a plain Transformer.

A design chosen by searching on real devices

Liquid AI’s LFM2 technical report (2025) shows how one lab arrived at a hybrid. It describes a hardware-in-the-loop search: candidate designs were scored on quality, speed and peak memory, with profiling run on a Samsung Galaxy S24 Ultra and an AMD Ryzen HX 370 laptop chip, and any design that broke the device budget was discarded.

The search kept landing on one layout. Nearly every layer is a short convolution with a gate that controls how much signal passes through, and only a handful of layers use grouped-query attention (GQA). In GQA, several attention heads share the same keys and values, which reduces memory traffic. The 1.2B model, for example, is listed with 6 attention layers among 16.

The report also says that adding linear-attention or state space operators did not improve aggregate quality and typically worsened device metrics. So a hybrid does not have to include Mamba-style layers. The report claims up to 2x faster prefill and decode on CPUs than similarly sized models. Those are Liquid AI’s own measurements.

We could not verify what the three components in LFM3’s tri-hybrid design are, and the details here should be taken from the model card linked in our roundup.

What hybrids give up

Fewer attention layers means less ability to recall exact details from a long input. Designs that keep too few can lose accuracy on tasks that depend on precise retrieval, so the ratio of attention to cheap layers is a tuned choice, not a free win.

There are practical costs too. Attention has years of optimized software and hardware support. Newer layer types may need custom code, and speedups measured on one chip may not carry over to another. Liquid AI’s report itself notes that mixture-of-experts variants still have room to improve on CPU.

Small hybrids also have stated limits. The LFM3-1.7B card, as summarized in our roundup, advises against agentic coding or complex multi-step reasoning, and its figures are vendor-reported. A lighter architecture changes cost, not the need for independent testing.

What to watch as hybrids spread

The pattern is now common across labs: a few expensive layers where they matter, cheap layers elsewhere. Watch for independent benchmarks at long context, since that is where hybrid memory savings should show most and where retrieval weaknesses would show first.

Also watch how far the ratio moves. Some designs keep roughly one attention layer in three, others use even fewer. Whether a standard recipe emerges, or whether each lab keeps tuning for its own target hardware, will shape how small models run on phones, laptops and edge devices.