agent-desktop vs AppleScript
432 runs of an AI agent operating Mac apps, on two macOS versions and three model sizes. The accessibility tree costs more, runs slower, and takes over your screen more often than AppleScript.
agent-desktop gives an AI agent "computer use without screenshots or pixel matching". It reads a desktop app's accessibility tree, hands the agent a JSON list of every control with a ref, and lets the agent click or type on a ref. Its actions are "headless by default: no pointer movement, no focus change". It runs on macOS, Windows and Linux.
I was wiring it into reminal, so that you could tap a button on your Mac from your phone, or watch an agent drive an app while you keep using the machine. Before building more on it, I wanted to know whether an agent actually does better with it. So I measured.
Short version: on macOS it did not. On two macOS versions and with three model sizes, an agent using agent-desktop was always slower and more expensive than one using screenshots or AppleScript. Its one edge, finishing a few more tasks than AppleScript with smaller models, came almost entirely from one app, and screenshots had the same edge at half the price. Working beside a person, which is the thing it promises, it took over the screen more often than AppleScript did.
The setup
The agent was Claude Code 2.1.317 running headless, with a fresh session for every run, on two Macs: macOS 13.6 in a Parallels VM, and macOS 26.6 in a VM on a Mac mini. It got the same task prompt in three setups:
- agent-desktop: the accessibility tree only, screenshots not allowed.
- Screenshots and clicks: a screenshot at 1280px, pixel clicks and key presses.
- AppleScript: no screen at all. AppleScript and the command line.
| Task | Prompt |
|---|---|
| Calculator | Compute 1234 × 56 and leave the result on the display |
| TextEdit | Create a new document and type "Quarterly report draft" |
| System Settings | Change the appearance to Light |
| Finder | Create a folder on the Desktop named Bench Reports |
| Safari form, Chrome form | Open a local sign-up form, fill it in, submit |
| VS Code | Change the color theme to Light Modern |
| Safari read | Open a local pricing page and report one plan's price and seats |
Each task ran three times per setup, in shuffled order. Claude Opus 5.5 did every task on both Macs; Claude Haiku 5.5 did them on both Macs too, and Claude Sonnet 5.5 on macOS 13.6. Then Opus did six of the tasks again while a simulated person typed on the same Mac. That is 432 runs. Success was checked by the harness after each run (the Calculator display, the document text, the setting, the folder on disk, the submitted page), never taken from the agent's own answer.
1. agent-desktop is the slowest and most expensive way to drive a Mac
With Claude Opus 5.5, every setup finished almost every task: agent-desktop 48 of 48, screenshots 48 of 48, AppleScript 44 of 48. What differed was the time and the money each task took.
| Setup | Time, 13.6 | Time, 26.6 | Cost, 13.6 | Cost, 26.6 |
|---|---|---|---|---|
| agent-desktop | 44s | 25s | $0.23 | $0.22 |
| Screenshots | 16s | 16s | $0.10 | $0.10 |
| AppleScript | 8s | 14s | $0.06 | $0.05 |
These are medians per task. agent-desktop got a lot faster on macOS 26.6 and was still the slowest. On both versions it cost about twice as much as screenshots and four times as much as AppleScript. The gap was widest in Finder and VS Code; the tables by app are below.
Time and cost by app (Claude Opus 5.5)
Median time:
| App, macOS | agent-desktop | Screenshots | AppleScript |
|---|---|---|---|
| Calculator, 13.6 | 9.7s | 15.3s | 7.6s |
| Calculator, 26.6 | 10.7s | 17.3s | 18.5s |
| TextEdit, 13.6 | 11.2s | 12.9s | 3.4s |
| TextEdit, 26.6 | 13.3s | 14.6s | 5.3s |
| System Settings, 13.6 | 14.3s | 15.5s | 8.4s |
| System Settings, 26.6 | 15.7s | 11.0s | 9.0s |
| Finder, 13.6 | 75s | 16.3s | 3.2s |
| Finder, 26.6 | 57s | 11.3s | 4.6s |
| VS Code, 13.6 | 215s | 16.3s | 18.7s |
| VS Code, 26.6 | 72s | 33s | 5.6s |
| Safari form, 13.6 | 64s | 24s | 38s |
| Safari form, 26.6 | 53s | 19.1s | 25s |
| Chrome form, 13.6 | 60s | 23s | 21s |
| Chrome form, 26.6 | 70s | 19.8s | 26s |
| Safari read, 13.6 | 10.7s | 8.1s | 3.3s |
| Safari read, 26.6 | 14.3s | 6.3s | 7.9s |
Median cost:
| App, macOS | agent-desktop | Screenshots | AppleScript |
|---|---|---|---|
| Calculator, 13.6 | $0.12 | $0.11 | $0.05 |
| Calculator, 26.6 | $0.11 | $0.10 | $0.06 |
| TextEdit, 13.6 | $0.15 | $0.11 | $0.06 |
| TextEdit, 26.6 | $0.05 | $0.06 | $0.01 |
| System Settings, 13.6 | $0.19 | $0.10 | $0.07 |
| System Settings, 26.6 | $0.14 | $0.05 | $0.03 |
| Finder, 13.6 | $0.50 | $0.15 | $0.07 |
| Finder, 26.6 | $0.36 | $0.06 | $0.01 |
| VS Code, 13.6 | $0.56 | $0.09 | $0.07 |
| VS Code, 26.6 | $0.29 | $0.15 | $0.02 |
| Safari form, 13.6 | $0.30 | $0.10 | $0.12 |
| Safari form, 26.6 | $0.32 | $0.15 | $0.11 |
| Chrome form, 13.6 | $0.30 | $0.10 | $0.05 |
| Chrome form, 26.6 | $0.40 | $0.10 | $0.06 |
| Safari read, 13.6 | $0.09 | $0.03 | $0.02 |
| Safari read, 26.6 | $0.11 | $0.03 | $0.02 |
AppleScript's VS Code times on 26.6 are for runs that did not finish; see section 4.
2. Why the accessibility tree costs more
No images, yet more tokens. I measured where they go. Every API call in a run log records its exact context size. The growth from one call to the next is whatever the previous tool output added, and every later call reads it again. Fitting the runs' costs gives the per-token prices exactly ($4 per million input, $8 per million cache writes, $0.20 per million cache reads, $20 per million output for Opus), so the tokens turn into dollars. This breakdown is for Opus on macOS 13.6.
| Extra cost of agent-desktop over screenshots | Per run |
|---|---|
| Accessibility snapshots: 7,178 tokens of tree text per run, against 4,485 tokens of images | +$0.070 |
| agent-desktop's own guide, read at the start: 6,278 tokens, carried on every later call | +$0.067 |
| More model output, because there are more steps | +$0.035 |
| Output of other agent-desktop commands | +$0.031 |
| Re-reading the system prompt on the extra calls (13 calls against 7) | +$0.024 |
| Screenshot images, which agent-desktop runs do not have | −$0.040 |
| Average per run: $0.277 against $0.104 | +$0.173 |
Refs expire whenever the UI changes, so the loop is snapshot, act, snapshot again to check. A screenshot agent often does several clicks from one look. More steps means the whole conversation gets read more times.
3. AppleScript already reaches the accessibility tree
This was the part I had not thought through. On macOS, System Events "UI scripting" (click button "7" of window 1 of process "Calculator") goes through the same accessibility API agent-desktop reads. The AppleScript agents used it whenever an app had no scripting support of its own: on macOS 13.6, 13 of 15 AppleScript steps in VS Code and 20 of 23 in the Safari form. For TextEdit and Finder they used the apps' own commands (make new document, make new folder), which is cheaper still.
So "AppleScript" here really means AppleScript including accessibility. That is why it held up even in VS Code on 13.6, at 19 seconds and seven cents against agent-desktop's 215 seconds and 56 cents.
4. A smaller model does not change the picture
A weaker model might get more out of a clean, structured tree, so I ran the same tasks with Claude Sonnet 5.5 and Claude Haiku 5.5. Tasks finished, out of 24:
| Model, macOS | agent-desktop | Screenshots | AppleScript |
|---|---|---|---|
| Opus, 13.6 | 24 | 24 | 23 |
| Sonnet, 13.6 | 23 | 24 | 21 |
| Haiku, 13.6 | 21 | 23 | 19 |
| Opus, 26.6 | 24 | 24 | 21 |
| Haiku, 26.6 | 22 | 23 | 24 |
agent-desktop finished more tasks than AppleScript in four of the five groups. Look closer and 10 of AppleScript's 12 misses are VS Code: the agent edited VS Code's settings file instead of using the app, and in this test VS Code ran with its own settings folder, so the edit never reached it. On a normal install that shortcut usually works, so this gap flatters agent-desktop. The one Haiku group that found the right folder, by reading how VS Code was launched, finished all 24.
Screenshots finished as many tasks as agent-desktop or more in every group. Median time per task:
| Model, macOS | agent-desktop | Screenshots | AppleScript |
|---|---|---|---|
| Opus, 13.6 | 44s | 16s | 8s |
| Sonnet, 13.6 | 28s | 11s | 5s |
| Haiku, 13.6 | 36s | 17s | 5s |
| Opus, 26.6 | 25s | 16s | 14s |
| Haiku, 26.6 | 22s | 15s | 11s |
With every model, agent-desktop was the slowest, 1.4 to 2.8 times slower than screenshots, and cost twice as much as screenshots or more. Smaller models are much cheaper per token (Haiku's list price is 40 times below Opus), so with Haiku the money gap shrinks to about a cent per task for agent-desktop against a fifth of a cent for AppleScript. The time gap does not shrink.
5. Working beside a person
agent-desktop's real promise is no pointer movement and no focus change, so an agent can work while you keep using your Mac. Screenshot agents cannot do that by design, so this test is agent-desktop against AppleScript.
A simulated person typed a known sentence into a TextEdit document for the whole run. The agent was told: someone is using this Mac and typing in TextEdit, do not bring other apps to the front, do not move the pointer, do not send keystrokes. The front app was sampled every 0.15 seconds. Six tasks, three runs each, with Claude Opus 5.5 on both macOS versions.
agent-desktop took the front away from the person more often than AppleScript did, on both versions, and cost five times as much per run:
| Measure | agent-desktop | AppleScript |
|---|---|---|
| Took the front, 13.6 | 7/18 | 3/18 |
| Took the front, 26.6 | 7/18 | 5/18 |
| Task done, 13.6 | 15/18 | 13/18 |
| Task done, 26.6 | 15/18 | 13/18 |
| Median cost per run | $0.38 | $0.07 |
In Calculator, System Settings and Finder neither setup took the front, apart from one Finder run, and both finished every time. In System Settings and Finder AppleScript cost 8 to 30 times less; in Calculator on 26.6 the two cost about the same ($0.12 and $0.15). The differences are all in the browsers and VS Code.
Working beside a person, by app
Runs where the agent took the front away from the person:
| App, macOS | agent-desktop | AppleScript |
|---|---|---|
| Calculator, 13.6 | 0/3 | 0/3 |
| Calculator, 26.6 | 0/3 | 0/3 |
| System Settings, 13.6 | 0/3 | 0/3 |
| System Settings, 26.6 | 0/3 | 0/3 |
| Finder, 13.6 | 0/3 | 0/3 |
| Finder, 26.6 | 1/3 | 0/3 |
| Chrome form, 13.6 | 3/3 | 3/3 |
| Chrome form, 26.6 | 3/3 | 3/3 |
| Safari form, 13.6 | 1/3 | 0/3 |
| Safari form, 26.6 | 0/3 | 2/3 |
| VS Code, 13.6 | 3/3 | 0/3 |
| VS Code, 26.6 | 3/3 | 0/3 |
Runs where the task got done:
| App, macOS | agent-desktop | AppleScript |
|---|---|---|
| Calculator, 13.6 | 3/3 | 3/3 |
| Calculator, 26.6 | 3/3 | 3/3 |
| System Settings, 13.6 | 3/3 | 3/3 |
| System Settings, 26.6 | 3/3 | 3/3 |
| Finder, 13.6 | 3/3 | 3/3 |
| Finder, 26.6 | 3/3 | 3/3 |
| Chrome form, 13.6 | 3/3 | 2/3 |
| Chrome form, 26.6 | 3/3 | 3/3 |
| Safari form, 13.6 | 1/3 | 2/3 |
| Safari form, 26.6 | 3/3 | 1/3 |
| VS Code, 13.6 | 2/3 | 0/3 |
| VS Code, 26.6 | 0/3 | 0/3 |
Median cost per run:
| App, macOS | agent-desktop | AppleScript |
|---|---|---|
| Calculator, 13.6 | $0.13 | $0.05 |
| Calculator, 26.6 | $0.12 | $0.15 |
| System Settings, 13.6 | $0.18 | $0.02 |
| System Settings, 26.6 | $0.17 | $0.02 |
| Finder, 13.6 | $0.41 | $0.02 |
| Finder, 26.6 | $0.60 | $0.02 |
| Chrome form, 13.6 | $0.37 | $0.22 |
| Chrome form, 26.6 | $0.41 | $0.39 |
| Safari form, 13.6 | $0.51 | $0.47 |
| Safari form, 26.6 | $0.64 | $0.43 |
| VS Code, 13.6 | $0.63 | $0.05 |
| VS Code, 26.6 | $0.34 | $0.02 |
The pointer never moved in any of the 72 runs. The logs show what took the front:
- Chrome, every run with both setups on both versions. agent-desktop's
launchbrings Chrome forward. The AppleScript agents activated Chrome or scripted it, despite the instruction. - VS Code, all six agent-desktop runs.
agent-desktop clickalone was enough, and VS Code stayed in front for 24% to 70% of the run. The no-focus-change promise did not hold for this Electron app. - Safari on 26.6, two AppleScript runs. Both agents were trying to switch on Safari's "Allow JavaScript from Apple Events" so they could fill the form with JavaScript. That setting lives in Safari's settings window and asks for confirmation; in one run I watched, the confirmation dialog was left sitting on the person's screen.
- Finder on 26.6, one agent-desktop run, where a click brought Finder forward for 11% of the run.
VS Code on macOS 13.6 is the one place agent-desktop finished where AppleScript could not, two times out of three, and it took over the screen to do it. On 26.6 it took over the screen and did not finish.
What I take from this
On a Mac, the accessibility tree is already within reach through AppleScript, and every model I tried knows how to use it. That was the fastest and cheapest way to drive an app, on both macOS versions. Screenshots were the most reliable across model sizes. agent-desktop sits between them on reliability and behind both on time and money. Its headless promise is the reason to want it, and in my runs it held for Apple's own apps and broke for Chrome and VS Code.
Where agent-desktop should still matter is Windows and Linux, which have no AppleScript. I did not test those.
Questions
What are these costs? Did you pay them? Each run's cost is what Claude Code reports at the end of the run: the API list price of every token the run used, including cache reads and writes. I checked it against the token counts for each model. The runs used a subscription, so nobody was billed these amounts. They work as a like-for-like price tag.
What exactly counts as a success? The harness checks the result after the agent says it is done: the number on Calculator's display, the text in the TextEdit document, the appearance setting, the folder on disk, the submitted form's page title, the theme in the settings file VS Code is actually using, the plan's price and seats in the answer. The agent's own claim of success is never used.
If agent-desktop sends no images, why does it use more tokens? Every step re-reads the whole conversation so far, and agent-desktop needs more steps: its refs expire when the UI changes, so it snapshots, acts, and snapshots again to check. It also starts by reading its own guide, about 6,000 tokens carried on every later step. And a snapshot of a busy window is longer than a screenshot: 7,178 tokens of tree text per run against 4,485 tokens of images. Section 2 breaks this down in dollars.
Doesn't AppleScript already have access to the accessibility tree? Yes. System Events UI scripting goes through the same macOS accessibility API that agent-desktop reads, and the AppleScript agents used it for apps without their own scripting support. That is the main reason AppleScript holds up so well here.
Was agent-desktop used the way it is meant to be used? The agent got the exact instruction agent-desktop's README suggests for an agent: before the first use, run agent-desktop skills get desktop and agent-desktop skills get platform and follow them. It was not allowed to fall back to screenshots, so every result comes from the accessibility tree.
Does a smaller model do better with the accessibility tree? Not compared with screenshots. Against AppleScript, agent-desktop finished more tasks in four of five groups, but almost all of that is VS Code (see the next question). On time and cost it was behind both other setups with every model.
Did the VS Code setup favor one approach? Yes, a little. To keep runs apart, the harness started VS Code with its own settings folder. Agents that changed the theme through VS Code itself, with agent-desktop or screenshots, were not affected. Agents that edited the usual settings file on disk changed a file this VS Code never read, and failed. On a normal install that shortcut would usually work, which would raise AppleScript's numbers.
So what is agent-desktop for? Its own pitch is computer use without screenshots, with no pointer movement and no focus change, on macOS, Windows and Linux. On macOS, AppleScript covered the first two in these tests, and the third did not hold for Chrome and VS Code. Windows and Linux have no AppleScript, so that is where it should matter. I did not test them.
Why were screenshot agents left out of the working-beside-a-person test? Clicking pixels moves the real pointer and typing goes to whatever app is in front. Taking over the screen is how that approach works, so there was nothing to measure.
Isn't three runs per cell too few? For the big gaps, no: they point the same way on both macOS versions and with every model. Small gaps, such as the Safari form costs, are within the run-to-run noise and should not be read as a difference.
Why two macOS versions? The first round ran on macOS 13.6, the version in my test VM. To check the result was not about an old macOS, I ran it again on macOS 26.6. The direction held; agent-desktop was faster on 26.6 but still the slowest.
Why no Sonnet on macOS 26.6? Time. Opus and Haiku bracket the range, and Sonnet on 13.6 landed between them.
How the numbers were kept honest
- Problems in the harness were fixed and every affected run was rerun: the form and pricing pages were not where the prompts pointed, a keychain dialog blocked four screenshot runs, and one run was hit by stray keystrokes from my own tooling. The tables hold only clean runs.
- On macOS 26.6, the Safari form check first failed because macOS had not yet allowed the harness to read Safari. Those runs were rerun once it was allowed.
- The first VS Code check also accepted the usual settings file, which the bench copy of VS Code does not read, so one run passed without changing the theme. The check now reads only the file VS Code uses, and every VS Code run that the old check had scored was rerun. On 26.6, an agent had also restarted VS Code without its settings folder; the reset now restarts it correctly before every run.
- On macOS 13.6, the AppleScript agents' edits to the usual VS Code settings file carried over from one run to the next, and one Haiku agent read the leftover value and stopped. That file is now reset before every run, and the Haiku VS Code runs were rerun.
- Three working-beside-a-person runs on 26.6 were rerun. Two were disturbed by my own clicks while I cleared a dialog an agent had left open. One failed only because that dialog blocked the check, although the form had been submitted.
- In the Safari read task the AppleScript agent opened the page in Safari but read the answer from the HTML file. The harness only checks the answer, so it counts as a success.
- The person meters were checked against controls on both macOS versions before use: a clean run read zero, forcing Calculator to the front read 37% away on 13.6 and 98% on 26.6, and a deliberate click registered as a pointer move. I also compared the person's document character by character, but some runs showed lost characters while their app never left the front, which I could not tie to a cause, so that measure is left out.
- Versions: Claude Code 2.1.317; Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5; agent-desktop 1.1.0; macOS 13.6 and 26.6; Chrome 156; VS Code 1.144.1.