cheliped-computer-use

Cheliped Computer Use (macOS)

Drive the local Mac from the terminal. One CLI takes a JSON array of commands and returns JSON — the same shape as the browser skill.

Output Envelope

Output is always a JSON array, one envelope per command, compact by default:

[{ "cmd": "screenshot", "ok": true,  "result": { "path": "...", "scale": 2 } },
 { "cmd": "type",       "ok": false, "error": { "code": "E_NO_PERMISSION", "message": "..." } }]

Each command runs even if a prior one fails (per-command isolation). Error code is one of E_NO_PERMISSION (Accessibility off), E_NO_CLICLICK, E_WRONG_FOCUS, E_DISABLED (escape hatch not enabled), E_BAD_ARG, E_UNKNOWN_CMD, E_TIMEOUT, E_UNKNOWN.

Flags (before the JSON arg): --pretty indents output; --allow-shell enables the run-shell / run-applescript escape hatches (off by default).

Setup

Built-ins (osascript, screencapture) need no install. Mouse control needs cliclick:

brew install cliclick

Permissions: the terminal/agent process must be granted Accessibility and Screen Recording in System Settings → Privacy & Security. The CLI now preflights Accessibility and returns E_NO_PERMISSION (instead of a silent no-op) when keyboard/mouse input would not be delivered; screenshots that come back near-uniform get a warning hinting at missing Screen Recording. Grant the permissions, then restart the terminal (permissions only apply to a freshly launched process).

Core Workflow: Observe-Act Loop

# 1. See the screen
node scripts/computer-cli.mjs '[{"cmd":"screenshot","args":["/tmp/now.png"]}]'

# 2. Act on what you saw (read the screenshot, then click / type)
node scripts/computer-cli.mjs '[{"cmd":"click","args":[640,400]},{"cmd":"type","args":["hello"]}]'

Read the screenshot with the Read tool, decide coordinates, then act. Re-screenshot to confirm.

Commands

Each row's Returns is the result object inside the envelope.