Tool
agent-desktop
vercel-labs/agent-desktop
What it is
A command-line tool and MCP server for computer use on Windows, macOS and Linux. It reads each desktop app's accessibility tree as JSON and lets an agent act on the element references it returns.
Who it might suit
Developers adding desktop control to an agent who prefer structured accessibility data over screenshot matching.
What to check first
It gives an agent control over desktop apps, and macOS needs an Accessibility permission grant. The README warns that snapshots, clipboard reads and screenshots can hold private content, and that the headed flag allows physical input.
How it runs
Command-line tool and MCP server
Written