Onscreen awareness: How an AI assistant understands what is on your screen

Onscreen awareness means an AI assistant can see what is currently displayed and act on it. You say “add this” or “what is this about?” and the assistant knows what “this” refers to, without you copying or describing anything. It turns a voice assistant from something that answers questions into something that works with the screen you are already looking at.

Apple’s HomeView smart display puts the idea on a kitchen counter. Apple says its Siri AI has onscreen awareness, meaning it can act on what is displayed.

An address in a text message, and one spoken sentence

The standard example comes from Apple’s own rollout. A friend texts you an address. Without screen awareness, you would open the message, copy the address, open Contacts and paste it. With it, you say “Add this address to their contact card,” and the assistant works out that “this address” is the one in the message on screen.

The same idea covers documents and web pages. While reading one, you can ask what it is about, and the assistant answers from the visible content. MacRumors described both examples in 2024 when Apple opened the feature to developers.

Two ways to know what the screen contains

An assistant can learn what is on screen in two broad ways. The first is to be told. The app that draws the screen hands the assistant a description of the content: this is a contact, this is an address, this is a document. Nothing has to be guessed, because the app already knows what it is showing.

The second is to look. A vision model takes the screen as an image and works out what the elements are. This works with any app, even one that never prepared for an assistant, but it is less exact. A model can misread a small icon or mistake one button for another.

Most real systems lean on the first route where possible and keep the second as a fallback or for research.

Apps that hand over their content

Apple’s approach, as reported in 2024, builds on App Intents, its framework that lets apps describe actions and content to the system. MacRumors reported that iOS 18.2 added a developer API through which an app can describe the content it is currently showing, so that Siri and Apple Intelligence can access it. The app marks what is visible, and the assistant can then retrieve it.

MacRumors also reported that a user can ask Siri about a document or website and, if the user explicitly asks, the content can be sent to supported third-party services. That detail matters: the feature is built around an explicit hand-off, not silent reading of everything displayed. Apple’s own privacy claims for the HomeView version are Apple’s and have not been independently verified.

The catch is coverage. The assistant can only act on what developers have exposed. An app that has not done the work is, to the assistant, mostly a blank.

Models that read pixels

The research route treats the screen as a picture to understand. In April 2024, Apple researchers Keen You, Haotian Zhang and colleagues published Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs (arXiv 2404.05719). A multimodal large language model (MLLM) is a language model that also accepts images.

The authors argue that general-purpose multimodal models often struggle with phone screens. Screens are tall and narrow, and their key elements, such as icons and text, are tiny. Ferret-UI uses an “any resolution” technique: each screen is split into two sub-images, horizontally for portrait screens and vertically for landscape ones. Each piece is encoded separately, so small details survive.

The model was trained on basic tasks such as icon recognition, finding text and listing widgets, plus reasoning tasks such as detailed description and inferring what a function does. The authors report it beat most open-source UI models and surpassed GPT-4V on the basic tasks, using a benchmark they built themselves. A follow-up, Ferret-UI 2 (arXiv 2410.18967), extends the work to iPhone, Android, iPad, web pages and Apple TV.

This paper does not say that Siri uses Ferret-UI. It shows the kind of capability pixel reading requires.

Where it breaks and what to watch

Screen awareness has real limits. Structured hand-offs only cover cooperating apps. Pixel reading can be wrong, and a wrong tap that sends a message or deletes a file costs more than a wrong answer. Assistants that act on screens tend to need confirmation steps for that reason.

Privacy is the other tension. A screen can show messages, banking details or health information. Any assistant that reads it has to decide what leaves the device, when, and with whose consent. Running on a device does not by itself mean nothing is sent elsewhere. Apple’s HomeView footnotes say that some tasks use server-based models, and that personal content depends on a paired iPhone or iPad.

Timing also matters. Apple announced onscreen awareness for Siri in 2024, but MacRumors reported that it was expected months later than the first Apple Intelligence release. HomeView shows the idea reaching a dedicated product with a shared screen, which raises new questions: whose content is on the display, and who can act on it.

Worth watching: how many third-party apps expose their content, whether assistants can read screens in apps that have not prepared, and how clearly each product states what is sent off the device.