Google’s test: A cheap classifier let Flash-Lite handle easy coding tasks

Google’s developer blog says AI coding agents should be judged by cost per successful task, not price per token. In a 50-question benchmark post, a small classifier routed tasks between Gemini models and solved all 50, but plain low thinking did too.

A classifier decides which Gemini model gets the task

The post, written by Sergio Villani of Google Cloud AI and dated Nov. 3, 2026, starts from a problem: per-token prices keep falling, yet bills for autonomous coding agents keep rising. Google’s explanation is that an agent which uses a high-thinking configuration for every small decision pays deep-reasoning prices for questions a lighter setup answers correctly. Its proposed fix is a routing layer that intercepts routine decisions before they reach an expensive model.

The example classifier is Jev from TypeSafe. It does not write text. It reads a structured question and answers in one pass, either by picking from a list of options, by giving a numeric rating, or by returning a Noul, a probability between 0 and 1 that a yes-or-no statement is true. Google states that it has no partnership with TypeSafe and that the post is not an endorsement. It adds that the same pattern can be built with other classifiers, including its own open-weights DiffusionGemma model, which a codelab runs in a customer’s Google Cloud project.

The routing recipe is simple. Every task goes first to Flash-Lite, which does not think by default. Jev then flags tasks that need more than one sequential step, and only those escalate to Gemini 3.8 Flash with thinking set to low.

One narrow question beat a broad one

Google reports that a single broad question, simple or complex, sent nearly everything to the simple path in an early test on 16 tasks. Replacing it with one atomic Noul question fixed the separation. Jev matched the hand-labeled difficulty on 13 of 16 tasks on the first run and 14 of 16 on the second.

The team then compared Jev with a free keyword-and-length heuristic in Python. The heuristic scored 15 of 16 (93.8%) on the questions used to write it. Frozen unchanged and tested on 34 unseen tasks, it fell to 70.6%. Long but easy prompts, such as pulling a port from a Kubernetes manifest, triggered false escalations. Short tricky ones, such as a Python dictionary with keys 1, True and 1.0, were sent down the simple path. Jev agreed with the labels on all 34, for 48 of 50 (96.0%) across the full suite, according to the post.

The post also warns against a common shortcut. Lowering maxOutputTokens on Flash did not reduce thinking on the 27 tasks Jev labeled simple, and it broke 7 of the 25 hard ones. Six ran out of room mid-reasoning, and one squeezed its reasoning and answered wrongly. The post recommends setting thinkingLevel: "LOW" directly.

The router’s rival was one setting

The most useful finding is a negative one. Flash on low thinking, with no router, also solved all 50 tasks, and Google calls it the number a router had to beat. The Jev router tied on accuracy. Google says it can reduce cost and response time against low thinking everywhere, though on prompts this short the cost gap per task is small and the shorter wait is the bigger benefit.

Flash-Lite on its own was fast on the simple tasks, at roughly half a second each, yet it got 6 of the 25 hard tasks wrong. Jev marked every one of those six as complex, with high confidence. The keyword heuristic was cheaper only because it failed 2 tasks. Escalating to Flash on default thinking cost more than low thinking everywhere. Google’s advice is to start with Flash on low thinking and build a router only if that falls short.

What the benchmark leaves unproven

Google lists its own limits: one run per setup, 50 prompts, hand-assigned difficulty and simple automatic grading. The figures are single-run measurements, and speed comparisons against default Flash or Pro mix two test rounds, so Google calls them approximate. The benchmark code will be published only after open-source review, so outsiders cannot yet reproduce the 50 of 50 result.

The prompts were also short and standalone, unlike many real agent turns. The post concedes that agents resend large blocks such as repository maps and tool definitions, and that Agent Platform Context Caching cuts the cost of resending them by 75% to 90%. Because a cache belongs to one model, switching between Flash-Lite and Flash can mean two caches or a lost discount. Google therefore advises routing only short, self-contained tasks.

Larger agent systems raise the stakes. Our explainer on orchestrator-worker AI notes that agents typically use about 4x the tokens of a chat and multi-agent systems about 15x, which is why routing is attractive in the first place. It also reports that AWS Kiro bills Claude Haiku 5.5 at a 0.06x credit multiplier, a sign that rivals are pushing the same cheap-model-first idea through pricing rather than classifiers.

One more caveat comes from the post itself: in Google’s run, the Gemini 3.1 Pro configuration declined two short code-security review prompts that Flash answered. Teams doing security reviews are told to test their own prompts on the configuration they plan to use. The post does not give per-task dollar figures in text, and its tables are published as images, so teams will have to run their own traces, ideally through Cloud Logging and BigQuery as Google suggests, before trusting any savings.