ByteDance’s Cola2 trains on 346B tokens, 60.1 average against 75.9 for Qwen3

ByteDance Seed has published Cola2 on Hugging Face, a 9-billion-parameter base language model that generates text by first producing compressed semantic latents with a diffusion transformer. It averages 60.10 across eight benchmarks after training on 346.03 billion tokens.

A diffusion transformer plans, a decoder writes

According to the model page, Cola2 combines three parts: a dynamic-compression VAE, a block-causal Diffusion Transformer (DiT) of about 4 billion parameters, and a span-causal token decoder that receives latent feedback. The idea is that the model works out meaning in a compressed space and then turns it into ordinary discrete tokens, rather than predicting one token at a time from the start.

Training has two stages. In the first, the encoder and token decoder are trained together. In the second, the VAE is adapted and the DiT is trained with Flow Matching on the latent distribution. The accompanying paper is titled “Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation” and is listed as arXiv 2610.31305.

The weights are a 9B-parameter checkpoint in F32 Safetensors format, split across eight files. Loading requires trust_remote_code=True, Python 3.10 or newer, PyTorch 2.2 or newer and a Transformers release from 4.51 up to the 4.x line. The page recommends CUDA with bfloat16. The default preset uses an 8-step Euler schedule, a DiT guidance scale of 10, a decoder guidance scale of -0.9 and max_new_tokens=32.

Output is not deterministic, because latent generation starts from Gaussian noise; users who want comparable runs need to pass a seeded torch.Generator. This is a base model, not an instruction-tuned assistant. The repository offers a chat script and a quick 200-example benchmark script. The page says no inference provider hosts it yet.

Numbers in context: 60.10 average on a fraction of the data

The page reports paper-based accuracy for eight tasks next to four open models. Cola2 trails all of them on the average, but it also used far less pretraining data.

• Cola2: 60.10 average, 346B tokens
• Llama-3-8B: 66.25, more than 15T tokens
• LLaDA-8B-Base: 72.11, 2.3T tokens
• Qwen2.5-7B: 75.08, 18T tokens
• Qwen3-8B-Base: 75.87, about 36T tokens

On individual tests the picture is mixed. Cola2 scores 41.75 on HellaSwag and 80.42 on StoryCloze, which the page says beats Llama-3-8B (39.21 and 79.49). It is well behind on MMLU (51.32 against 72.31 for Qwen3-8B-Base) and on RACE (69.91 against 90.65). It also falls below every comparison model on LAMBADA (63.00) and TruthfulQA (54.10).

The page itself cautions that the models use different training recipes and tokenizers, so the table is context and not a controlled comparison. Each Cola2 figure is the mean of three evaluations with different latent noise. Nothing here has been independently reproduced, and the data gap makes the headline average hard to read in either direction: a smaller budget explains part of the shortfall, but the page does not show what Cola2 would score with more tokens.

Technical significance: A rival to masked diffusion

The closest reference point on the page is LLaDA, which the authors treat as a baseline. According to its paper, submitted in February 2025, LLaDA is a diffusion language model trained from scratch that masks tokens in a forward process and predicts them back with a Transformer. Cola2 takes a different route: it does not diffuse over tokens but over continuous latents, then decodes.

The page includes a controlled scaling study. It says the Cola-MDM variant shows the strongest aggregate scaling, while Cola2-MDM converges more slowly but keeps narrowing its gap to LLaDA after roughly 150B tokens. It adds that the best decoder guidance value stays near -0.9 for most of training and drifts toward zero. Those are the authors’ observations, and the underlying figures are linked from the project pages rather than reproduced here.

ByteDance Seed has been busy on the research side this week. It also released three 17 GB sparse autoencoders for Qwen3’s mixture-of-experts layers, which are interpretability tools rather than a new language model. Cola2 sits in a different part of the research agenda: changing how text is generated, not explaining how it is stored. For background on the interpretability side, see our explainer on sparse autoencoders.

Open code, but weights have separate terms

The repository code is under the Apache License 2.0. The page says the model weights, tokenizer assets, benchmark datasets and third-party materials are subject to their own terms, with attribution details in a NOTICE file. The page as read does not spell out what those weight terms allow, so commercial users should check the repository before building on it. No individual authors are listed on the model page.

For developers, Cola2 is a research artifact: it is a base model with a short default output length, requires remote code, and lags the strongest similarly sized models on knowledge-heavy tests. Its value is as a working example of latent-space text generation that others can run and probe. Whether the approach closes the gap at larger data budgets is not established by the source.