Computer-use agents: How AI models learn to click, type and run a shell

A computer-use agent is an AI model that operates ordinary software the way a person does. It looks at the screen, decides what to do, and then moves the pointer, clicks or types. Some also run commands in a terminal. No special connection to each app is needed.

Picture asking an assistant to copy a table from an email into a spreadsheet and chart it. A human opens the email, selects the table, switches windows and builds the chart. A computer-use agent tries to do the same sequence, one step at a time.

Looking, acting, and looking again

The core of these systems is a loop. The model receives a screenshot and the user’s goal. It outputs one action, such as click at this spot or type this text. A small program carries out that action on a real or virtual machine. Then a fresh screenshot goes back to the model, and the loop repeats until the task is done or a step limit is reached.

The model has to be a vision-language model, meaning it can read images and text together. It must also work out where things are on the screen. Researchers call that GUI grounding: linking an instruction like “open the Export menu” to the exact pixels of the right button. GUI stands for graphical user interface, the windows and buttons people use.

Why not just use an API?

Many programs have no clean programmatic interface, and each integration takes work to build. A model that can use the screen can, in principle, use any software made for people. Anthropic described this reasoning when it released computer use in public beta in October 2024: instead of building tools for individual tasks, it taught Claude general computer skills.

The cost is speed and fragility. Every step needs a new screenshot and a new model call. One wrong click can derail the rest of the task.

From a 12% benchmark to a product category

The standard yardstick is OSWorld, introduced in April 2024 by Tianbao Xie and colleagues. It contains 369 tasks across real Ubuntu, Windows and macOS environments, including web and desktop apps, file handling and workflows that span several programs. Each task ships with a script that checks whether the result is correct, so scoring does not depend on a judge’s opinion.

In the paper, humans completed over 72% of tasks, while the best model managed 12.24%. The authors blamed weak GUI grounding and a lack of operational knowledge, meaning knowing how software is actually used.

Products followed quickly. Anthropic reported 14.9% on OSWorld for Claude 3.5 Sonnet in the screenshot-only setting, and 22.0% with more steps allowed. It called the feature experimental and at times error-prone, and noted that scrolling, dragging and zooming were hard. In January 2025, OpenAI introduced its Computer-Using Agent, which powers Operator. OpenAI describes it as combining GPT-4o’s vision with reasoning trained through reinforcement learning, and it reported 38.1% on OSWorld.

Adding a terminal: The Mage-VL-Cowork approach

Clicking is not always the best tool. Editing a hundred spreadsheet cells by mouse is slow, while a short script does it at once. Newer agents therefore mix two action channels. Microsoft’s open Mage-VL-Cowork, a 4-billion-parameter model, has one channel for GUI actions and another for a persistent shell and Python environment. Microsoft says the model chooses a channel on every turn, without an outside router.

Microsoft reports 66.0% on OSWorld-Verified, an updated and corrected version of the benchmark, which it says is the best among 4B models. Its own table also lists larger and proprietary systems that score higher, plus a human reference of 72.4%. Those figures are Microsoft’s, and the model card does not point to independent reproduction. It does include an evaluation harness so others can test the claim.

The same release shows how training has changed. Microsoft describes five stages, ending with online reinforcement learning, where the model practices in sandboxed machines and is rewarded when a task checker says it succeeded. According to the card, that final stage raised the OSWorld-Verified score from 55.84% to 66.04% while cutting average output tokens per task.

Limits worth knowing before trusting one

Benchmark numbers hide several caveats. Scores depend on the step budget, the number of runs averaged and which tasks are excluded. Microsoft’s 66.0%, for example, is a two-run average on 361 tasks with a 50-step budget. Results from different labs are not always directly comparable.

Context is another constraint. Screenshots are large, so agents must forget or compress older turns. Mage-VL-Cowork keeps only the three most recent screenshots and folds earlier steps into short summaries.

Safety is the larger issue. An agent that can click and run commands can also do damage if it is fooled by hostile text on a web page or an email. Anthropic noted when launching computer use that it could be a route for spam or fraud, and advised starting with low-risk tasks. The Mage-VL-Cowork card, as read by our news report, gives little detail on safeguards for an agent with shell access.

What to watch next

Three things are worth tracking. First, whether small open models keep closing the gap with large proprietary ones, since a 4B model is cheap enough to run locally. Second, independent reruns of OSWorld-Verified results rather than vendor tables. Third, how agents are sandboxed and permissioned, because the practical question is not only whether an agent can finish a task, but what it is allowed to touch while doing it.