computer-use-mcp: Instructions file for Codex

Using computer-use-mcp with AI Agents

This guide covers how to integrate computer-use-mcp into AI agent frameworks and agentic workflows. Works on macOS, Windows, and Linux.

Skills (recommended): copy from skills/ in this package into your agent skills directory:

• skills/computer-use — when/how to use desktop control
• skills/computer-use-forms — accessibility form fill
• skills/computer-use-scripting — AppleScript / PowerShell first
• skills/computer-use-recovery — FocusFailure recovery
• skills/computer-use-windows-admin — filesystem / registry / process
• skills/blender-agent — Blender through its own MCP, this server only for what has no API

MCP prompts: diagnose-desktop, fill-form, script-first, safe-desktop-task
Profiles: COMPUTER_USE_PROFILE=core|desktop|ax|scripting|windows-admin|full (default desktop since 7.6: core plus the OCR and wait tools and agent_pointer; full lists everything)

Current surface (7.7.0):

• Default list. On macOS the default desktop profile lists 37 tools and 44,324 bytes on the wire (about 11k tokens); COMPUTER_USE_PROFILE=full lists 78 tools and 82,698 bytes (112,727 before 7.6). 7.7.0 added run_steps and lists the click aliases (left_click, double_click and the rest) only in full; they stay callable in every profile. Tool counts by profile on macOS: core 33, desktop 37, scripting 40, windows-admin 47, ax 54, full 78 (Windows 31/33/36/45/50/76, Linux 31/33/36/37/50/68). A test derives these from the catalog and checks this file, the README and the computer-use skill. Byte budgets are under test (desktop 48,000, full 88,000).
• Only tools that exist here are listed. The catalog gives each tool platforms and a job (observe, act, semantic, script, admin, browser, spaces, meta). registry and notification are Windows-only; read_window_text, click_text, wait_for_text and wait_for_stable are macOS-only; the Spaces tools are macOS and Windows. A tool that does not exist on the running platform is neither listed nor callable (before 7.6 it was listed and returned platform_unsupported).
• One click. click {coordinate, button: left|right|middle, count: 1|2|3} plus the usual targeting and delivery. left_click, right_click, middle_click, double_click and triple_click remain callable in every profile as thin aliases of it (listed only in full since 7.7); they still accept target_app, target_window_id, delivery and force. Examples here use click.
• Every action says what happened (7.7). Each action's result carries effect (verified, changed, no_visible_change, unverified), route and, for input, delivery (hid or pid). Every refusal and every unverified result adds a hint and a next_call: one read-only call this session can make as is. verify: true on an input tool captures before and after. A JSON reply gains these as fields; a text reply keeps its text first and gains a JSON block; both are in structuredContent.
• run_steps (7.7). Up to 30 steps ({tool, args}) in one call: every step validated before any runs, the target focused once, stopped at the first failing step (the rest reported not run), one optional capture at the end. Steps are input, accessibility actions, activation, write_clipboard and the waits; scripts and admin tools keep their own call.
• Session target. set_target {app | window_id | title | clear} and get_target make the target explicit, so input and capture tools can omit targeting; screenshot full_screen: true captures the whole screen even when a target is set.
• Waits. wait_for_window, wait_for_text (OCR: text appears, or gone: true) and wait_for_stable (the window stops changing) replace sleep-and-screenshot loops. wait is still there for a plain delay.
• Server instructions are generated per platform and profile from the catalog (src/instructions.ts). They give the route order (the app's own API, scripting, accessibility, OCR for self-drawn apps, coordinates last), the user-active guard and delivery, the waits and the session target. They no longer tell hosts to prefer the accessibility tree, which is empty on Unreal, Blender and games. get_tool_guide(task) picks the route for a task.
• Arguments sent as text (7.7). "[100, 200]", "100,200", "500" and "true" are accepted where an array, a number or a flag is declared, but only when the original fails its schema and the coerced value passes.
• Requirements. Node 20 or later. macOS 14 or later for the ScreenCaptureKit capture and the OCR tools. The user-active guard reads physical input through an event tap that needs Input Monitoring for the host app; without it the guard allows every call and doctor says its clock is unavailable.