What It Does

agent-desktop gives an AI agent the ability to use a computer, often called computer use. It reads the accessibility tree of a desktop app and returns it as JSON, where every actionable element carries a ref. The agent clicks or types on a ref, then reads the UI again to verify, and the loop is kept by the agent itself. The tool contains no model. It ships both as a command line interface, CLI, and as a C library, covering macOS, Windows, and Linux, with roughly 60 commands. In short, it turns desktop apps into a set of buttons and input fields that a program can call.

The Problem It Solves

Traditionally, getting an AI to operate a GUI means taking a screenshot and handing it to a vision model. That is costly, slow, and brittle against small UI changes. agent-desktop switches to a structured, lightweight accessibility tree, so the agent gets a semantically clear element tree instead of pixels, which is more reliable, cheaper, and faster. It also exposes every command as an MCP tool through agent-desktop mcp, so it drops straight into mainstream agent frameworks. For contrast, when a human drives a remote machine the usual route is remote desktop; see How to Use Windows Remote Desktop (RDP). Here the operator is the AI instead, which raises the automation level.

Language and Architecture

  • Main language is Rust.
  • The core reaches the desktop only through each system's native accessibility API via PlatformAdapter: AX API on macOS, UI Automation on Windows, and AT-SPI on Linux, with both X11 and Wayland supported.
  • The same core drives both the CLI and the in process C FFI library libagent_desktop_ffi, which other programs can call directly.
  • It ships an MCP server over stdio, with tool names mapped one to one to commands, such as desktop_snapshot and desktop_click.

License

Apache-2.0

In 2026, computer use, meaning agents that drive the desktop, is a major hotspot, and this Vercel project offers a more stable and faster engineering approach than screenshot based methods. It runs on three platforms and ships MCP adapters, gaining about 3,791 stars that day and topping the Trending list. Vercel's reach in frontend and deployment also helps it get picked up by developers and agent frameworks quickly.

https://github.com/vercel-labs/agent-desktop (site https://agent-browser.dev/desktop)

Summary

agent-desktop pulls computer use back from the screenshot path to a steadier structured path. For teams that want to give their agents desktop control, it removes the pain of reverse engineering the UI and maintaining a vision model, and that makes it a component worth watching.

Getting Started

To try it locally, first confirm your agent framework supports MCP, then wire in agent-desktop mcp and let the agent call tools like desktop_snapshot and desktop_click. On the first run, verify permissions in a test window before pointing it at important apps. Accessibility permission is granted manually in each OS, with macOS accessibility and Windows UI Automation each having their own entry. In production, scope operations to specific apps to cut the risk of mistakes. Read the tool list once so you know what the agent can do, and start with read only snapshots before allowing clicks, which keeps the early experiments safe while you learn the model's behavior.

FAQ

Accessibility tree versus screenshots, what is the real win? A screenshot gives pixels that break on font, theme, or resolution changes, and recognition burns compute. The accessibility tree gives you semantics: what each button is called and where it sits. That is clearer to act on and cheaper to run.

Can a normal user use it today? It is low level, aimed at developers and agent frameworks, not an app you install and click. To try it you need to run a CLI and wire its MCP into whatever agent you use.

Does it touch system permissions? It calls the OS native accessibility interface, which usually requires you to grant permission manually, like accessibility access on macOS. Think about the scope before granting, and avoid turning it on a public machine.

Is the C library useful on its own? Yes. If you already have a native app or another language runtime, the FFI library lets you embed desktop control directly without shelling out to the CLI, which keeps latency low and integration tight.