computer-use

Background computer use (macOS)

You have access to a Codex-style computer-use tool surface that drives macOS applications in the background — without activating them, bringing them to the front, or moving the user's cursor. The tools are provided by the background-computer-use MCP server and are named:

• list_apps
• get_app_state
• click
• drag
• perform_secondary_action
• press_key
• scroll
• set_value
• type_text

The only interaction pattern that works

Use this loop every time:

1. Discover (optional). If you don't know which app to drive, call list_apps. The app argument on every other tool accepts either the display name ("Finder") or the bundle identifier ("com.apple.finder").
2. Observe. Call get_app_state(app) once at the start of every turn before acting on that app. Its response contains:
   • an indexed accessibility tree of the key window
   • a window screenshot
   • (sometimes) bundled app-specific hints under <app_hints>
3. Decide. Pick element indexes from the tree wherever possible. Those indexes are valid only until your next get_app_state call — treat them as snapshot-local, never durable.
4. Act. Invoke click / type_text / press_key / scroll / set_value / drag / perform_secondary_action using the index you just picked (or raw coordinates as a fallback).
5. Re-observe after any action that likely changed the UI (menu opened, view navigated, sheet presented, async result arrived). Stale indexes are the #1 cause of silent failures.

If you skip step 2, tools that take element_index will fail with "No captured app state for this app."

Accessibility vs. coordinates

Prefer accessibility-based targeting. It is more robust to window moves, scaling, and theme changes.

• Use click(app, element_index=N) when the target appears in the tree. For left single-clicks on elements that support AXPress, the server calls AXPress directly, which works even when the app is occluded.
• Use click(app, x=..., y=...) only when the tree does not surface the target (e.g. custom canvases, WebViews without AX, games). Coordinates are screen points in the top-left origin.
• Use perform_secondary_action(app, element_index, action) when the element advertises an action like "open", "Increment", "Decrement", "Pick", "ScrollToVisible", "Confirm". Most secondary actions are background-safe.
  • Refused by default: AXShowMenu, AXShowAlternateUI, and AXRaise almost always bring the target app to the foreground because they need to present a visible menu or modal. The tool raises ToolError on these unless you pass allow_foreground_activation=True. Only use that flag when you've confirmed with the user that activating the app is acceptable for this turn.