Accessibility APIs: How one program can read and click another app
Accessibility APIs give screen readers a labeled map of an app's buttons and menus, and let other programs press them. Here is how that works and why AI agents touch it.

An accessibility API is a built-in channel through which an operating system lets one program read the interface of another and act on it. Screen readers use it to speak a window aloud. Automation tools use it to press buttons without a human. The channel is the reason a new class of AI agents asks for an “Accessibility” permission.
Anthropic’s computer use beta for Team and Enterprise plans is one example. On macOS it requires that permission.
A labeled map behind every window
Picture a blind user opening a mail app. The screen shows a toolbar, a list of messages and a Send button. Pixels alone tell a screen reader nothing. So each app also publishes a second, invisible description of itself: a list of every element, with a name, a type and a state.
For the Send button, that description says roughly: this is a button, it is called “Send,” it is enabled, and it can be pressed. The screen reader reads the description aloud and can trigger the press on request.
That description is the accessibility tree. It is a map, not a picture.
How the tree is built and used

The tree has a root, then applications, then windows, then smaller parts. Each element carries a few kinds of information:
- A role or control type, such as button, text box or menu item.
- Properties, such as the label, the current value and whether it is enabled.
- Actions a client can request, such as pressing or selecting.
- Events, which tell a client when something changes, such as a new window gaining focus.
Apple’s archived accessibility guide describes the macOS version this way. The system-wide element is the root, the application is its child, and the menu bar and windows hang below. The guide says the protocol defines only a handful of actions. Apps keep control: a client asks, and the app decides what “press” does.
Microsoft’s framework, UI Automation (UIA), works similarly. Its documentation says the tree is rooted at the desktop, and that clients can use a filtered control view or content view to skip clutter. It adds control patterns, such as an invoke pattern for buttons and a scroll pattern for lists, which expose type-specific methods. A combo box, for example, supports at least expand and selection patterns.
From Windows 95 to today
Microsoft Active Accessibility (MSAA) is the older Windows interface. Microsoft’s documentation says it was introduced in Windows 95. Each element is an IAccessible object with a role, a state and a few fixed actions such as accDoDefaultAction. Microsoft notes that interaction is limited to what that interface offers, and that its tree contains many irrelevant elements that clients must filter.
UI Automation was the richer successor. Microsoft’s overview says it first appeared in Windows XP as part of the .NET Framework, and that the API was rewritten on the Component Object Model (COM) for Windows 7. Wikipedia dates the original release to 2005 and puts the native API in the Windows Vista era, so sources disagree on the exact first date. A built-in bridge lets old and new clients talk to each other.
Apple’s model, built around the NSAccessibility protocol, serves the same purpose on macOS. Apple’s archived page is dated 2015 and does not state when the model was introduced.
The design goal was never AI. It was access for people who cannot use a screen and mouse. Test automation and robotic process automation (RPA) piggybacked on the same channel, because a program that can read the tree and press buttons can also run a script.
Where AI agents come in
A computer-use agent has two broad ways to see an app. It can look at a screenshot and guess where to click, or it can read the tree and press a named element. Each has strengths. The tree gives exact labels and exact targets. A screenshot works on any app, including ones that publish no tree.
Anthropic’s developer documentation for its computer use tool describes the screenshot route. Claude receives images, returns actions such as left_click, type and key, and gives coordinates in screenshot pixels. The documentation does not mention accessibility APIs in the parts we could read. It does warn that Claude “might make mistakes or hallucinate when outputting specific coordinates,” and that reliability can be lower with niche applications or several applications at once.
The release notes for the Team and Enterprise beta say something separate. On macOS, Claude Desktop needs both Accessibility and Screen and System Audio Recording permissions. A management profile can pre-grant Accessibility, but Anthropic says it cannot switch on Screen Recording. Windows needs no operating system permission. Anthropic’s pages, as we read them, do not explain exactly what the Accessibility permission is used for, so any link to the tree remains an inference rather than a stated fact.
The pairing is still instructive. On macOS, the permission for synthetic input and for reading other apps is the same gate that screen readers pass through. Granting it is a significant decision.
Limits, risks and what to watch
The tree is only as good as the app that publishes it. Apps that label controls poorly give clients poor maps. Custom-drawn interfaces and unusual widgets can be hard to describe. Microsoft says MSAA exposes less information than UIA, and Apple’s guide shows each app deciding which elements to hide.
Permissions are the second limit. The accessibility channel is powerful by design. Anything allowed to read and press another app’s controls can, in principle, reach whatever that app shows. For that reason, operating systems put it behind explicit user consent. Microsoft’s documentation also notes that UIA does not enable communication between processes started by different users through Run as.
The third issue is oversight. Anthropic’s own setup guidance, as summarized in our news story, says there is no sandbox between Claude and a user’s applications, and that screenshots are saved with the chat. Administrators can block named apps or set a computerUseEnabled policy key to false, though Anthropic says the key is not a lock against someone who can change the computer’s settings.
What to watch next:
- Whether agent vendors document how much they read from the tree versus the screen.
- Whether operating systems add finer permissions, separating “read the interface” from “press buttons.”
- Whether app makers improve their labels, which would help screen reader users and agents alike.
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Daniel ReedAI Research, Safety & Ethics AnalystDaniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…



