Vercel releases agent-desktop: agents control any app without screenshots

If you have ever tried to automate a desktop application with a screenshot-based model, you know the pain: it is slow, expensive and brittle. A single shifted pixel or a change of theme, and the agent misses the button. Vercel Labs proposes a different path: agent-desktop, a tool that does not look at the picture at all.

How it works

The operating system already knows what is on the screen. For screen readers it has an accessibility tree: a tree of elements with roles, names, states and focus. On macOS this is the AX API, on Windows UI Automation, on Linux AT-SPI. agent-desktop reads this tree and hands the agent JSON in which every element has a ref.

The loop is simple. The agent requests the tree, finds the button it needs, clicks it by its ref, reads the tree again and checks that the action worked. For example, instead of "click roughly at point 540×320", the agent says "press ref e12, this is the Send button" and confirms that the input field has been cleared.

Under the hood is a single Rust binary of about 15 MB with no dependencies. It is installed with the command npm install -g agent-desktop. There are 60 commands, identical on all three platforms; each prints JSON and exits. There is no LLM inside: the "look, act, verify" loop lives in your agent. The license is Apache-2.0, and there is also a skill that the agent reads before it starts working. For Electron and Chromium, the tool opens a DevTools port, which agent-browser can then connect to.

The main problem: context

A full application tree is huge. A Slack dump can easily weigh more than 50,000 tokens, and the agent drowns before the first click. So a "skeleton" a couple of levels deep is fetched first, and only the needed branch is expanded. According to the author, on Slack, VS Code and Notion this saves 78–96% of tokens. These figures are the author's own and have not been independently verified yet, so measure on your own application.

The project's history is also interesting. It began as the work of a single developer inspired by agent-browser, and now the tool is released under the Vercel Labs brand and sits in their documentation. Either the person was hired or the project was acquired. A classic open-source path.

Limitations

The main limitation follows from the idea itself: if an application does not publish an accessibility tree, the agent cannot see it. Custom-drawn controls, games and some Electron builds with accessibility turned off will slip through. On Windows there is a separate trap: if a window runs with elevated privileges, the tool runs into a security policy.

This matters especially for enterprise software. That is exactly where, in 1C, accounting packages and internal clients, automation is needed most, and accessibility support is often rudimentary. Loud claims about the death of RPA should be taken with a grain of salt: first check that your application exposes a tree at all.

What to do in practice

The analogy with Playwright is accurate: structure instead of pixels. If you have internal desktop systems, spend an hour and look at their accessibility tree through agent-desktop. If it is rich, you get automation that is noticeably cheaper and more stable than computer use on screenshots. If it is empty, screenshots remain the fallback.

Product people and founders should note one more thing: the tool is not tied to a particular agent harness. It is simply a CLI that can be plugged into any agent, and that is its strength. Start with one routine task, for example exporting a report from an old client, and compare token cost and the share of successful runs with your current approach.

This is not the end of RPA, but it is a serious step: agents now have a cheap, portable way to see an interface the way the system itself sees it.