GUI grounding: How an AI agent works out where to click on a screen
GUI grounding is the skill of matching a plain-language instruction to the exact button, field or icon on a screen. It decides whether a computer-use agent clicks the right thing.

GUI grounding is the ability of an AI model to link a plain-language instruction, such as “press Save”, to the exact spot on a screen where that button sits. GUI stands for graphical user interface: the windows, buttons and menus people click every day. An agent that cannot do this step reliably cannot operate software, however well it plans.
A person tells you to “open the blue Export button”
Picture a colleague standing behind you, pointing at your monitor. You say, “Click the gray gear icon in the top right.” They scan the picture, find the gear and put a finger on it. They never read the program’s source code. They just look.
A vision-based agent does the same job. It receives a screenshot, reads the goal, and must answer one question: where exactly should the cursor go? The answer is usually a pair of numbers, the horizontal and vertical position of the click. Getting those numbers right is grounding. Choosing which task to do next is a separate skill, called planning.
Google’s new Mac feature, which we covered in Gemini can now run your Mac apps while you keep working, depends on this step. Google’s release notes describe an agent that clicks, types and fills in forms across desktop apps and websites. They do not describe, in the material we reviewed, how Gemini locates each element internally.
Two ways to see a screen: A map of labels or a picture of pixels

Software can describe a screen to an agent in two broad ways. The first is structured text. Operating systems and browsers expose a list of every button and field with a name and role, often called the accessibility tree. Clicking then means choosing an item from the list. This is precise, but apps differ in how well they fill it in, and some, such as games or custom canvases, expose almost nothing.
The second way is the raw screenshot. The model looks at pixels, like a person would, and outputs a position. It works on any app that can be drawn on a display. The price is that the model must recognize tiny icons and dense text, then convert its judgment into accurate coordinates.
Many systems mix both: structure where it exists, pixels where it does not.
From text-heavy photos to clicking models: The research record
General vision-language models, which describe photos well, were weak at this. Screens are crowded with small text and icons, unlike a typical photo of a dog on a beach. In 2024, a team led by Kanzhi Cheng published SeeClick (ACL 2024). It trained a vision-language model to predict click and typing locations from screenshots alone, without parsing a page’s HTML. Click points were expressed as ratios of the image size, for example 0.49 across and 0.40 down.
The same paper introduced ScreenSpot, a benchmark of over 600 screenshots and 1,200 instructions covering iOS, Android, macOS, Windows and web pages. A model is scored on whether its predicted point lands inside the right element. The authors reported that gains on this test carried over to downstream agent tasks.
Larger datasets followed. The UGround work trained on about 10 million GUI elements from 1.3 million screenshots, according to its paper.
Parsing the screen first: The OmniParser approach
A second line of work splits the problem in two. In August 2024, Microsoft Research authors Yadong Lu, Jianwei Yang, Yelong Shen and Ahmed Awadallah published OmniParser. A detector first finds the clickable regions in a screenshot and draws labeled boxes around them, and a captioning step describes each one. The language model then picks a label such as “box 14” instead of inventing coordinates.
The idea is to make the language model’s job easier by handing it a numbered menu rather than a blank image. The team also released a dataset of interactable regions whose ground-truth boxes came from webpage DOM trees, the structure behind a web page. Microsoft later released OmniParser V2, described as turning any language model into a computer-use agent. Its GitHub page reports 39.5% on the ScreenSpot-Pro benchmark for V2.
Where grounding still fails, and what to watch
Benchmarks reveal the limits. ScreenSpot-Pro was created because, according to its authors as summarized in follow-up literature, the original ScreenSpot used cropped local regions rather than full, realistic screens. Professional software has very high resolutions and tiny controls, so scores there are far lower than on the simpler test. V2’s 39.5% on that harder test shows how much room remains.
The broader picture comes from OSWorld, a 2024 benchmark of open-ended tasks in real computer environments. Its authors reported that humans completed about 72% of tasks, while the best model at launch reached 12.24%. A later efficiency study reported a best agent at 42.5% success. The project moved to a corrected OSWorld-Verified version in July 2025, so older and newer scores are not directly comparable. Epoch AI’s review argued that about 10% of tasks have serious errors and that many can be done with little GUI use, so the scores do not isolate clicking skill.
Grounding errors matter more as agents get more control. A click that lands one element off can send the wrong file or confirm the wrong form. This is why products such as Google’s add guardrails: per its release notes, users choose which apps the agent may use, it pauses for logins and payments, and it may ask for confirmation before sensitive actions.
Things worth watching are scores on full-resolution, professional-software tests, whether agents combine accessibility data with pixels, and whether vendors publish grounding results for their own products. Today, most such figures come from research models, not shipping apps.
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Daniel ReedAI Research, Safety & Ethics AnalystDaniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…
You may also like

Apple’s new MacBook Pro adds touch and a 10,000,000:1 OLED screen, from $2,899

Reward hacking: How AI models learn to win the test without doing the task

Terminal-Bench: How a command-line exam scores AI agents on real work
