Your agent uses the apps on your Macwhile you keep typing

Ask your agent for something you would otherwise click through in a Mac app. It reads the window as a list of named controls, acts on them, and shows you in the conversation exactly what changed. It works on windows behind the one you are using, so Preview, Slack, System Settings or your own desktop build are in reach without the agent taking your screen.

claude · ~/leases
> Sign the lease in Preview with my saved signature. Don't save or send it.
$
claudeSigned the lease on page 2 with your saved signature. I did not save or send it; that part is yours.

What you can hand it

“Sign the lease in Preview with my saved signature.”

Clicks Sign, picks the saved signature from the popover, and stops before saving or sending.

find · click · action

“Check that the desktop build shows the new settings pane.”

Reads your app's own window, opens the pane by name, and reports the tree and a screenshot.

get-app-state · click · wait · shot

“Pick the export folder in the save dialog.”

Works the native file picker and permission sheets a web page cannot reach.

find · set-value · press-key

“Is Bluetooth on, and which mic is selected?”

Reads System Settings and answers. Reading is the agent's to do; changing a setting waits for you to ask.

find · get-app-state

The agent works in windows behind yours

Every verb works on a background window. The agent's keys go to the app it is driving, never to the one you are typing in. No verb brings a window forward on its own.

The orange pointer is the agent. Any action that lands on a point draws it gliding there and pulsing on the press, over the screen and without taking focus, so you can see what is happening without being interrupted. --no-cursor hides it for one action.

When an action does need a real mouse event, your pointer returns to where it was. The one flag that raises the target window is --restore-window, and agents are told to pass it only when nothing else will do.

set-value, perform-secondary-action, click on a pressable element
background window
via accessibility.Works on a background window and can be verified.
type-text, press-key, hotkey
background window
via the app's own event queue.Keys go to the target app, never to the app you are typing in.
click --x --y on a background window
background window
via accessibility, at that point.Presses the control under the point without moving your pointer.
paste-text
background window
via the clipboard.Your clipboard is restored afterwards. Capped at 16 MiB.
click --mouse, drag, a control with no press
needs the window in front
via a real mouse event.macOS drops a press on a background window, so this fails with window_not_focused. --restore-window brings it forward, and your pointer goes back where it was.

How the agent reads a window

An agent names an app and gets back one window as text: every element on its own line, with an index, a role, a name and the actions it advertises. In the conversation it shows as the step's output, with a picture of the window beside it once Screen Recording is granted. This is a real Finder tree, with the file names changed.

Documents
Search
Favorites
Recents
Applications
Documents
Downloads
Name
Lease renewal.pdf
Offsite agenda.key
Invoice 0931.pdf

Point at the window or at a line of the tree.

Printing part of the tree. find "Sign" prints only the matches and their ancestors. --under 31 prints one subtree. Both take a /regex/.

Gaps in the numbering. The tree drops noise, so indexes skip. An agent never counts its way to one; it reads the number off the line.

Old indexes fail. An index belongs to the tree it came from. A stale one fails as element_not_found instead of clicking whatever sits there now.

$ cast computer get-app-state --app com.apple.finder --no-screenshot
Finder (pid 749, com.apple.finder)
Window: id:1514 "Documents" (1070x902 @ 375,32)
Visible elements: 64 Focused: #31 Coordinates: window
Truncated: no
Screenshot skipped (--no-screenshot)
0 standard window Documents
1 split group
2 scroll area, Secondary Actions: scroll up, scroll down
3 outline sidebar
7 outline row Recents
8 cell Recents, Secondary Actions: open
10 outline row Applications
11 cell Applications, Secondary Actions: open
13 outline row (selected) Documents
14 cell (selected) Documents, Secondary Actions: open
16 outline row Downloads
17 cell Downloads, Secondary Actions: open
24 scroll area
31 list Documents
32 row Lease renewal.pdf
35 row Offsite agenda.key
38 row Invoice 0931.pdf
40 toolbar
41 button (disabled) Back
44 search field Search
The focused UI element is 31 list Documents.

What each action reports back

An action prints one sentence saying what was attempted, by which route and whether it was verified, then the lines of the tree that moved. The agent gets the indexes for its next step without taking another snapshot, and finds out right away when an app ignored it.

the action's own output
$ cast computer click --app com.apple.Preview --element "Sign"
Click attempted via accessibility (AXPress), unverified (accessibility action unasserted).
Changes: 3 added, 0 removed
+ 88 popover
+ 89 button Created January 27, Secondary Actions: insert signature
+ 90 button Create Signature
with --json
"verification": { "state": "unverified", "reason": "accessibility_action_unasserted" }

macOS cannot confirm that a press did what a button promises, so a click reads attempted. The diff under it is the proof. It shows a popover with three new elements, and the indexes to use next.

Grant the two macOS permissions

macOS asks a person to grant Accessibility and Screen Recording by hand. Until then the agent cannot read a window, and says so. Codecast signs a small helper app and asks for both on its behalf, so neither Codecast nor your terminal holds them.

The Computer card on the Agent features page: a Preview window read as numbered lines with the agent's orange pointer on button Share, the New and On badges and the card's switch, and the line Agents can use native Mac apps for you, in the background.

The setup page grants both from the browser, or in Agent features, switch on the Computer card. It lists the two grants still missing, and its Open System Settings button opens the pane for the next one. When both land, the card reads Permissions granted.

In the desktop app, Settings, Desktop, Permissions shows them under Agent computer use as Computer control and Computer screenshots. If an agent reaches for an app before they are granted, the same steps appear as a card under its step in the conversation, with Tell the agent to continue once they are done.

1
Accessibility

Lets the helper read the window you name and act inside it. Every verb needs it. It goes to the helper and not to your terminal, because macOS attaches a permission to the program that asks, and your terminal runs everything you and every agent type there.

2
Screen Recording

Lets the helper take a picture of the window it just read, so an image comes back beside the text. The text works without it; only the picture fails.

From a terminal, cast computer setup walks through both grants the same way, and cast computer permissions reads them without opening anything.

Linux hosts

On Linux under X11 the same verbs run through AT-SPI and a small helper the CLI writes to ~/.codecast/computer/linux/. It answers the same protocol as the Mac helper, so trees, diffs and filters read the same. --app takes the window's WM_CLASS there, such as google-chrome.

Limits that protect your Mac

Your Mac holds your mail, your bank and your passwords. These limits are enforced where they cannot be talked around, or written into the instructions every agent reads.

Password managers are refused

The helper itself refuses these, under any name an agent tries. No setting or instruction turns it off.

1PasswordBitwardenDashlaneLastPassNordPassProton Pass

Secret fields never print

A password, passcode or one-time code field shows as [redacted] in every tree, so its value never reaches the agent's context or your transcript.

14 text field Email, Value: [email protected]
15 secure text field Password, Value: [redacted]

Secrets go in through stdin

--text-stdin and --value-stdin keep a token out of shell history and ps. Passing both a flag and stdin, or stdin from a terminal, fails with an error.

$ printf '%s' "$TOKEN" | cast computer set-value --app <app> --element-index 42 --value-stdin

The agent reads on its own and waits for you before changing anything

Every agent with codecast installed carries the same standing instruction: do not push, submit a form, send a message, buy anything, delete data or change account settings unless you asked for that action. In an app holding sensitive content, read only what you were asked to read.

agent does
waits for you to ask
✓ read a window
✕ send a message
✓ find a button
✕ submit a form
✓ write a draft
✕ buy anything
✓ take a screenshot
✕ delete data
✓ report what it saw
✕ change settings

Run several steps in one command

This section is about the agent's speed. Each separate command pays a second or more to start the CLI. cast computer do runs a whole flow against one app over one helper connection. It stops at the first failing step and reports what ran and what never did; --keep-going carries on.

one process, one helper connection
$ cast computer do --app com.apple.Preview - <<'EOF'
click "Sign"
wait "Created January"
action "insert signature" "Created January 27"
press Return
shot
EOF
what it prints (abbreviated)
✓ click "Sign"
Click attempted via accessibility (AXPress), unverified (accessibility action unasserted).
✓ wait "Created January"
✓ action "insert signature" "Created January 27"
Secondary action attempted via accessibility (insert signature), unverified (accessibility action unasserted).
✓ press Return
✓ shot

Steps read the way the window does, and each takes the single command's flags. A bare click or action after a find acts on what it found.

snapshotfindwaitclickactionsettypepastepresshotkeyscrolldragshotsleep

Error codes and their recoveries

A failure carries a code and the recovery for it: in --json as code and recovery, printed under the message otherwise. All of them say to change something before retrying and never to rerun a command unchanged. These are the recoveries word for word.

▸app_not_found
  1. Run cast computer list-apps and retry with the exact bundle id.
  2. A web app such as Gmail is not an app selector: target the desktop browser that contains it.
  3. If the browser is not listed, open or focus it first.
▸app_blocked
  1. Stop. Do not drive this app.
  2. Choose another target or ask the human to do it.
▸window_not_found
  1. Run cast computer list-windows --app <app> and target a listed window.
  2. If the app is listed with no visible window, retry observation once with --restore-window.
  3. cast computer does not launch closed apps.
▸window_not_focused
  1. Mouse input reaches only a window in front; keys and typing reach a background one.
  2. Look for a route that needs no mouse: a keyboard shortcut, a menu item, a Secondary Action, or set-value.
  3. If only the mouse will do, pass --restore-window, which takes the human's screen; if the message says presses were already delivered, run get-app-state first.
▸window_stale
  1. Run cast computer list-windows --app <app> and choose a current selector.
  2. Then rerun get-app-state before acting.
▸element_not_found
  1. Run cast computer get-app-state --app <app> again and use an element index from the fresh tree.
  2. Never infer an index from elementCount.
  3. Never reuse an index after navigation, scrolling, a focus change or a delay.
▸element_not_clickable
  1. Choose a parent or child that has a frame.
  2. Or use window-local coordinates taken from the latest screenshot.
▸action_not_supported
  1. Read the element's Secondary Actions in a fresh snapshot and use one of those names.
  2. Or use click or set-value.
▸value_not_settable
  1. Choose a settable element from a fresh snapshot.
  2. If none accepts a write, focus it and use keyboard input, then inspect the returned state.
▸invalid_argument
  1. Fix the flags exactly as the message says.
  2. Do not retry unchanged.
▸action_timeout
  1. Run get-app-state first so you know whether the UI changed.
  2. Then retry with a simpler semantic action, or with --no-screenshot if observation is slow.
  3. Do not repeat the same timed-out action blindly.
▸permission_denied
  1. Run cast computer permissions to read the grants; it reports and shows nothing on screen.
  2. To grant, ask the human first: cast computer permissions --open-settings --id accessibility takes the front.
  3. A token or peer failure means the helper belongs to another user or another launch: rerun the command, which relaunches it.
▸screenshot_failed
  1. If the tree is enough, rerun with --no-screenshot.
  2. If the message names Screen Recording, cast computer permissions --open-settings --id screenshots opens the pane. It takes the front, so ask the human first.
▸unsupported_capability
  1. Run cast computer capabilities and choose a supported action.
  2. Or use a semantic alternative such as set-value or click.
▸provider_incompatible
  1. Run cast computer capabilities.
  2. Update codecast; the CLI and the helper it materializes must come from the same release.
▸accessibility_error
  1. Run cast computer capabilities.
  2. If the message names Accessibility, cast computer permissions --open-settings --id accessibility opens the pane. It takes the front, so ask the human first.
  3. If it names the helper app, run cast doctor.
  4. Do not loop while availability is unchanged.

Command reference

Agents learn these from their instructions; you only need them to script it yourself. From cast computer --help. cast computer help <verb> prints one verb's flags from the binary about to run them, so an agent never works from a stale list.

Set up
setup
Walk a human through both grants. Idempotent; nothing opens until they confirm.
permissions
Read the Accessibility and Screen Recording grants. Shows nothing on screen.
capabilities
What the helper supports on this machine. Sets it up on first run.
Look
list-apps
Running apps with a bundle id and a pid.
list-windows
Visible windows of one app, with the id and index other verbs target.
get-app-state
The indexed accessibility tree of one window, plus a screenshot.
find <text>
Only the matching elements, with their ancestors.
wait [text]
Until text appears, disappears (--gone) or anything changes (--change).
Act
click
By index, by name or at a window-local point.
perform-secondary-action
Run one of an element's advertised Secondary Actions.
set-value
Write a value straight into an element and read it back.
type-text
Type into the focused element of a background window.
press-key / hotkey
One key, or a modifier chord like CmdOrCtrl+A.
paste-text
Paste through the clipboard, restoring what was there.
scroll / drag
Scroll an element or point; press, move and release.
Batch
do [steps...]
Many steps against one app in one process. - reads them from stdin.
help <verb>
That verb's flags, straight from the binary about to run it.
Flags that matter
--app <selector>
Bundle id (preferred), app name, or pid:1234. On Linux, the WM_CLASS.
--window-id / --window-index
One window of an app with several, from list-windows.
--element / --element-index / --nth
Act by name (several matches are listed, never guessed) or by index.
--find <text> / --under <index>
Print part of the tree. /regex/ works too.
--diff
get-app-state prints only what changed since this window's last read.
--json
Machine-readable. Errors carry code and recovery; actions carry action.verification.
--screenshot / --no-screenshot
Snapshots capture by default, actions do not.
--restore-window
Raise the target window first. The only flag that takes your screen.
--no-cursor
Hide the agent cursor for one action. CODECAST_COMPUTER_CURSOR=0 hides it everywhere.
--text-stdin / --value-stdin
Read a secret from stdin, out of shell history and ps.
--keep-going
With do: carry on past a failing step.

Limits

Which platforms?
macOS, and Linux under X11 through AT-SPI. Both need a desktop session with the app open. There is no Windows provider.
Does it open apps?
No. It drives windows that are already open. A closed app answers window_not_found with that said plainly.
What about apps that draw their own UI?
A canvas, a game or a drawing exposes little to accessibility, so the tree says little. Coordinate clicks and screenshots still work, and the CLI says when a change may sit outside what the tree shows.
Is “attempted” a failure?
No. It means the helper delivered the action but could not read back proof, which is normal for a press. The diff under it is the evidence; verified appears only when the change was read back.
When does it need my screen?
Only for a real mouse event: --mouse, drag, or a control with no accessibility press. macOS drops those on a background window, so the command fails with window_not_focused unless the agent passes --restore-window.
Is this for web pages too?
Inside a page, agents use the browser: it holds your logins and speaks the page's own structure. Computer is for native apps and for what a page cannot reach, such as the address bar, a file picker or a permission sheet.
Taking longer than usualOffline · reloads when back