Terminal-Bench: How a command-line exam scores AI agents on real work

Terminal-Bench is a benchmark that tests whether an AI agent can do real technical work in a command-line terminal. The agent gets a task, a sandboxed computer and no hints. Automated tests then check whether the job was actually done, so the score reflects results rather than how convincing the agent’s answer sounds.

A test where the computer is the exam room

Imagine hiring a freelance engineer and saying only: “Build this software from its source code and make it run.” You would not grade an essay about how they might do it. You would run the program and see whether it works.

Terminal-Bench grades agents the same way. The agent types commands, reads the output, edits files and tries again, much as a developer does. When it stops, a hidden set of checks looks at the final state of the machine. Either the program runs, the file is correct and the model is trained, or the task counts as failed.

This is different from a quiz of questions and answers. It measures whether an agent can plan, use tools, recover from errors and finish.

What one task contains

According to the paper, each task has three parts: its own environment, a solution written by a human, and thorough tests that verify the result. The environment is the starting state of the machine, with the files and software the task needs. The human solution proves that the task can be solved at all. The tests decide pass or fail.

The authors describe the tasks as inspired by real workflows and as demanding high-skill work. Examples they give include training a machine learning model, building and running Linux from its source code, and reverse engineering binary files (compiled programs with no readable source).

A terminal is the text window where people type commands to a computer. Working there is a natural fit for agents. Everything is text, and almost any software task can be done through it.

Where it came from and who runs it

The Terminal-Bench 2.0 paper, “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces,” was posted to arXiv in January 2026 and accepted at ICLR 2026, a major machine learning conference. Mike A. Merrill, Alexander G. Shaw and Nicholas Carlini are listed as equal-contribution corresponding authors, with a long list of other contributors. The first two affiliations are Stanford University and the Laude Institute.

The authors say the motivation was a gap in existing benchmarks. In their view, some do not measure real-world work, and others are not hard enough to challenge frontier models, meaning the most capable systems available. The 2.0 release has 89 curated tasks.

The project’s website says it is hosted by Stanford, Harbor and the Laude Institute, with source code on GitHub. Tasks run on Harbor, an evaluation framework named in the paper. The paper also describes a reference agent called Terminus 2, which lets different models be compared on the same footing.

What the scores say, and how Google used them

The 2.0 paper reports that frontier models and agents resolved fewer than 65% of tasks, and that smaller models scored around 15%. The authors also ran an error analysis to show where models and agents fall short. The headline measure is the resolution rate, the share of tasks fully solved. The current leaderboard on tbench.ai reports it for version 4.1, with 95% confidence intervals, and also lists cost and tokens used.

The benchmark appears in our report on Gemini 4 Hydrogen. Google says Hydrogen scores 57% on Terminal-Bench 4.1, against 18.2% for Gemini 3.8 Flash. Google also says its Orchestrator feature in Antigravity, which switches between models inside one session, reached near-Carbon success rates on Terminal-Bench 4.0 while using 24% less quota and finishing 19% faster. These are Google’s figures, and they have not been verified independently.

Note that the two claims use different versions, 4.1 and 4.0. Tasks and difficulty can change between versions, so the numbers should not be compared directly.

Reading a Terminal-Bench number with care

A pass-or-fail check on a real machine is hard to charm, but it is not flawless. Several limits matter for readers.

• Setup affects results. The same model can score differently depending on the agent wrapped around it, the time allowed and the commands it can run. A score belongs to a model plus a harness, not to the model alone.
• Tests can be gamed. An agent might find a shortcut that satisfies the checks without doing the job. The paper’s quality-control section mentions an adversarial exploit agent, which suggests the authors treat this as a real risk.
• Versions move. Scores on 2.0, 4.0 and 4.1 are separate scales.
• Cost counts. Two agents with the same resolution rate can use very different amounts of compute, which is why the leaderboard shows cost and tokens as well.
• Tasks are a sample. A few dozen hard tasks cover a narrow slice of professional work, so the confidence intervals matter.

What to watch next

Three things are worth following. First, whether newer versions keep their difficulty as models improve, since a benchmark that most agents pass stops being useful. Second, whether labs report results on the same agent harness, which would make vendor claims easier to compare. Third, whether cost and speed become part of the headline, as in Google’s quota and time claims for Orchestrator, rather than a footnote to the success rate.

For now, a Terminal-Bench score is best read as a useful signal about how well agents handle real technical work, as long as the version, the harness and the source of the number are all stated.