agent-desktop Lets AI Agents Use Your Desktop Apps Without Looking at the Screen

Vercel Labs' new tool reads apps the way a screen reader does. Its best idea is that every button reference is tied to one snapshot, so a stale click fails instead of hitting the wrong thing.

Most AI agents that "use a computer" are squinting at it.

They take a screenshot, guess where the Save button is from the pixels, and send a click to a coordinate. It works often enough to demo. It also fails in a very specific, very annoying way: a dialog slides in, the layout shifts by forty pixels, and the agent confidently clicks the thing that used to be there.

Your computer already has a cleaner way to describe what is on screen. It is the accessibility tree, the structured list of windows, buttons, text fields and labels that screen readers use to tell blind users what an app contains. Every major desktop OS has one. Agents have mostly ignored it.

agent-desktop, which sat at the top of Trendshift's daily board on October 31, is built on that tree. It is a command-line tool and C library for macOS, Windows and Linux that "reads the accessibility tree of a desktop app and returns it as JSON," in the words of its docs. Version 1.0.2 shipped on October 30, shortly after the repository moved to the Vercel Labs organization.

The headline feature is not that it reads the tree. Plenty of accessibility tools do. The interesting part is a small design decision about references that makes desktop agents fail honestly, and one permission detail you should get right before you install it.

What it is, in plain terms

agent-desktop is not an agent. The docs say so directly. It is the hands and eyes an agent calls.

Your agent (Claude Code, Codex, your own script) runs a command. agent-desktop prints one block of JSON and exits. No window, no chat box, no background brain. The agent reads the JSON, decides what to do, runs another command, and checks the result.

That loop is the whole product:

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

The first command asks Finder for its accessibility tree and keeps only the elements you can interact with (that is what -i does, per the README). Each one comes back with a ref, a short label the agent can point at. The second command clicks one. The third looks again to see what changed.

That last step is not optional politeness. The docs put it plainly: "A successful action does not prove the UI changed, so the loop ends with a read."

The idea worth stealing: refs tied to a snapshot

Look at the ref in that example: @s8f3k2p9:e8. The part before the colon is the ID of the snapshot that produced it. The part after is the element.

That prefix is the clever bit. A ref only means something relative to the snapshot it came from. A bare @e8 is rejected with an INVALID_ARGS error. And if the app has changed since the snapshot (a dialog opened, a list re-sorted), an action can come back with STALE_REF or AMBIGUOUS_TARGET instead of doing something. Refs to elements that have not moved can keep working; the point is that a ref the tool can no longer match cleanly gets rejected rather than guessed at. The agent's job is then simple: take a new snapshot and use the new refs.

Compare that with the screenshot approach. A pixel agent that clicks at (812, 440) after the layout moved does not get an error. It gets a click on whatever is at (812, 440) now. You find out later, from the side effects.

A good everyday comparison is a numbered ticket at a deli counter. The number is only valid for that day's queue. Show up with yesterday's ticket and nobody hands you someone else's order. They tell you to take a new one.

That is the shift in one line: desktop automation stops being "click where the button was" and becomes "click this button, from this view, or tell me the view is out of date."

Two more design choices back it up:

Every command returns the same envelope. It has ok, plus either data or an error with a code, a disposition and usually a suggestion. The README tells agents to retry only when error.disposition.retry is safe. That gives an agent a clear rule for "try again" versus "stop and think," which most automation tools leave to guesswork.
Actions are headless by default, meaning they run in the background. Ref actions do not move your mouse pointer or steal focus. You can keep typing in another window while the agent works. If an app only responds to real input, the global --headed flag switches to physical clicks and keystrokes.

Big apps without drowning the agent

An accessibility tree for something like Slack is enormous. Dumping all of it into an agent's context would waste tokens and bury the useful parts.

agent-desktop's answer is a skeleton first, then a drill-down:

agent-desktop snapshot --skeleton --app Slack -i --compact
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact

The first command maps the app shallowly, like reading a table of contents. The second expands one region, using a ref from the skeleton as the starting point. It is how a person scans a busy screen: find the sidebar, then look inside it.

Put this into practice

Here is the lowest-friction way to try it this weekend.

1. Install it.

npm install -g agent-desktop
agent-desktop version

Bun, Yarn and pnpm work too, and there are archives on the releases page. Official Homebrew and Scoop packages are still being submitted, per the README.

2. Grant permissions (and read the next section first).

On macOS, run agent-desktop permissions --guide and grant Accessibility access when prompted. Windows needs no extra grant for apps at the same privilege level as your terminal. Linux needs a graphical session with at-spi2-core running.

3. Teach your agent the commands.

The project ships an agent skill (a packaged set of instructions an agent loads on demand):

npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

There is also an AGENTS.md snippet that tells agents to run agent-desktop skills get desktop before first use, which prints docs matched to the installed version.

4. Start with something read-only.

Ask your agent to snapshot a harmless app, say Calculator or a notes app, and describe what it sees. Then have it click one button and verify. You will see the ref format, the envelope and the verify step in a few minutes, without risking anything.

5. Then try a real chore.

Good first targets are boring, repetitive desktop tasks with no web API: exporting a report from a desktop finance app, renaming files through a GUI-only tool, filling a form in an internal Electron app (a desktop app built from web technology, the way Slack and VS Code are). For Chromium and Electron apps you launch, the docs say agent-desktop can open a DevTools port that its sibling tool, agent-browser, connects to for the web content inside.

The permission you should get right

On macOS, the Accessibility grant goes to "the app that runs agent-desktop," per the docs. In practice that is usually your terminal.

That is the detail worth slowing down for. Accessibility access is powerful by design: it lets software read and operate other apps. Grant it to your everyday terminal and you have granted it to the terminal, not to one tool.

The practical move is a dedicated terminal app that exists only for agent work. Grant Accessibility there and keep your main terminal without it. A second profile inside the same terminal app does not count; the grant belongs to the app, so every profile in it shares the permission.

Two related cautions from the README itself. Snapshots, screenshots and clipboard reads "can contain private content," so whatever is on screen can end up in your agent's context and, from there, at your model provider. And an optional helper the project calls JEV sends screen descriptions to a configured model provider. Leave it off until you know you need it.

Honest limitations

Here is what it will not do, in plain words.

It can only see what apps expose. No pixels means no fallback. If an app publishes a thin or broken accessibility tree (some games, some custom-drawn tools, plenty of older software), agent-desktop sees a thin or broken app. The docs are clear on this: it is "limited to what apps expose."

Not every command works yet. 56 of 60 commands are operational. key-down, key-up, mouse-down and mouse-up refuse to run everywhere until there is a daemon that can release a held key or button. The higher-level drag command does work, so this only rules out workflows that need a key or button held down by hand.

Platform gaps are real. The capability table varies by OS. On Linux, screenshots work for X11 windows while Wayland goes through a portal, and the Wayland clipboard needs wl-clipboard. Multiple windows of the same app cause AMBIGUOUS_TARGET until you pass --window-id.

It is young. Version 1.0.0 and 1.0.1 both shipped on October 28 and 1.0.2 two days later. Expect churn. The v1.0.2 notes also mention a new warning for npm users when the Node launcher slows each call, which suggests the native archive may feel faster.

Small housekeeping. The LICENSE file is Apache-2.0, but its copyright line is still the unfilled template placeholder. It does not change what you can do with the code. It does say something about how fresh the project is.

The bigger point

Desktop agents have been held back less by model intelligence than by eyesight. Guessing at pixels is slow, expensive and wrong in ways that are hard to notice.

agent-desktop bets that the better path was sitting in every OS all along, built for screen readers. Its most useful contribution is a habit, not a feature: every reference belongs to a snapshot, every action gets verified, and "the screen changed" becomes an error instead of a mystery. You can steal that pattern for your own tools even if you never install this one.

If you do install it, give it its own terminal, start with a harmless app, and watch what it reads. That first snapshot will tell you more about how ready your apps are for agents than any benchmark.

Sources: vercel-labs/agent-desktop README · agent-desktop releases (v1.0.2, October 30, 2026) · agent-desktop docs · Trendshift daily board, read about 07:20 ET October 31, 2026. Reviewed version: v1.0.2.