Test-time compute: How letting a model think longer improves its answers

Test-time compute is the processing power a language model spends while answering a question, as opposed to the power spent training it. Letting a model “think” for more steps before it replies can make its answer better, at the price of more waiting and more tokens. Reasoning-effort settings such as low, medium and high are a dial on exactly this.

Think of a math exam. A student who must blurt out each answer in two seconds will miss problems that the same student would solve with scratch paper and ten minutes. The student did not get smarter; they got room to work. Test-time compute gives a model that scratch paper.

Scratch paper made of tokens

A language model writes its output one token (a word fragment) at a time. Each token costs one pass through the network. A reasoning model is trained to write a long stretch of intermediate steps first, often called a chain of thought, and only then the final answer.

Those steps let the model break a problem into parts, check a result, notice a mistake and try another route. The longer the working-out, the more compute the answer used. That is why “thinking longer” and “spending more test-time compute” mean nearly the same thing.

Training matters too. According to OpenAI, reinforcement learning teaches its o1 model to refine its strategies, recognize and fix mistakes, and switch approach when one is not working.

Some systems also spend compute differently: they sample several candidate answers and pick the best one, or use a second model to score partial solutions. All of these trade extra work at answer time for accuracy.

A dial you can turn per request

Because the extra thinking is optional, vendors expose it as a setting. Anthropic’s documentation describes two generations of this control. Its older “extended thinking” mode takes a budget_tokens value, with a minimum of 1,024 tokens, as a target for how much the model may reason before answering. The docs say the budget is a target rather than a strict cap, and that larger budgets can improve quality with diminishing returns.

Anthropic now marks that mode as legacy. Newer models use “adaptive thinking”, where the model decides whether and how much to think, steered by an effort setting. At lower effort it may skip thinking entirely on easy inputs.

Mistral offers the same idea in its model. According to Mistral’s release post, as summarized in our report on Le Chonk, the model has three reasoning levels (low, medium and high, with high the default) so users can trade depth for cost.

Where the idea came from

The research thread is recent. In August 2024, Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar posted “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters” (arXiv:2408.03314). They studied two ways to use extra compute: searching with verifier models that score reasoning steps, and letting the model revise its own answers.

Their main finding was that the best method depends on how hard the prompt is. A “compute-optimal” strategy that adapts per prompt improved efficiency by more than 4x over a best-of-N baseline, where the model simply samples N answers and picks one. In a comparison matched for total compute, a smaller model with non-trivial success on a problem could beat a model 14x larger.

On September 12, 2024, OpenAI described o1 in “Learning to reason with LLMs”. OpenAI reported that o1 improves both with more reinforcement learning during training and with more time spent thinking at answer time. Its charts on the AIME math competition plot accuracy against compute on a log scale. ARC Prize’s analysis read that as a log-linear pattern: accuracy rises steadily only when compute grows by multiples.

Who pays, and how much

Thinking is not free. The reasoning tokens are generated by the model and billed as output. Anthropic’s docs note that its API reports how many of the billed output tokens were internal reasoning, so developers can see the cost. On Le Chonk’s API, output costs $2.10 per million tokens, according to Mistral, so a long reasoning chain on every request adds up.

Longer thinking also means longer waits. Anthropic advises batch processing for thinking budgets above 32,000 tokens, because very long requests can hit timeouts and open-connection limits.

This is why token efficiency has become a selling point. Mistral says Le Chonk scored 48% on the Artificial Analysis Coding Agent Index, effectively tied with GLM-5.3, while using 47% fewer tokens per task. If that holds up, a model reaches a similar score while spending less test-time compute. These are Mistral’s figures and have not been independently verified.

Limits and what to watch

More thinking does not always help. Easy questions gain little, and Anthropic’s docs describe diminishing returns that vary by task. A longer chain of thought is also not proof that the reasoning is sound. ARC Prize argued that more test-time compute alone does not settle the path to general intelligence, and that future work should look at scaling search and refinement more efficiently.

Three developments are worth following. First, adaptive control: models that choose their own thinking depth rather than relying on a fixed budget. Second, cost per solved task, which compares models more fairly than raw benchmark scores when one spends far more tokens than another. Third, how well open-weight models match closed ones at each effort level, since users who run those themselves can set the dial freely.

When comparing two models, check the effort setting used in each benchmark. A high result from one lab and a low result from another are not a like-for-like test, and the token count behind a score is part of the story.