macos-computer-use

macOS Computer Use (universal, any-model)

You have a computer_use tool that drives the Mac in the background. Your actions do NOT move the user's cursor, steal keyboard focus, or switch Spaces. The user can keep typing in their editor while you click around in Safari in another Space. This is the opposite of pyautogui-style automation.

Everything here works with any tool-capable model — Claude, GPT, Gemini, or an open model running through a local OpenAI-compatible endpoint. There is no Anthropic-native schema to learn.

The canonical workflow

Step 1 — Capture first. Almost every task starts with:

computer_use(action="capture", mode="som", app="Safari")

Returns a screenshot with numbered overlays on every interactable element AND an AX-tree index like:

#1  AXButton 'Back' @ (12, 80, 28, 28) [Safari]
#2  AXTextField 'Address and Search' @ (80, 80, 900, 32) [Safari]
#7  AXLink 'Sign In' @ (900, 420, 80, 24) [Safari]
...

Step 2 — Click by element index. This is the single most important habit:

computer_use(action="click", element=7)

Much more reliable than pixel coordinates for every model. Claude was trained on both; other models are often only reliable with indices.

Step 3 — Verify. After any state-changing action, re-capture. You can save a round-trip by asking for the post-action capture inline:

computer_use(action="click", element=7, capture_after=True)

Capture modes

mode	Returns	Best for
som (default)	Screenshot + numbered overlays + AX index	Vision models; preferred default
vision	Plain screenshot	When SOM overlay interferes with what you want to verify
ax	AX tree only, no image	Text-only models, or when you don't need to see pixels

Actions

capture           mode=som|vision|ax   app=…  (default: current app)
click             element=N     OR     coordinate=[x, y]
double_click      element=N     OR     coordinate=[x, y]
right_click       element=N     OR     coordinate=[x, y]
middle_click      element=N     OR     coordinate=[x, y]
drag              from_element=N, to_element=M        (or from/to_coordinate)
scroll            direction=up|down|left|right   amount=3 (ticks)
type              text="…"
key               keys="cmd+s" | "return" | "escape" | "ctrl+alt+t"
wait              seconds=0.5
list_apps
focus_app         app="Safari"  raise_window=false   (default: don't raise)