computer-use

Computer Use (desktop automation)

When to use

• Native desktop apps, installers, modal dialogs, simulators
• Apps that draw their own UI (Unreal, Blender, games): OCR route below
• UI-only workflows with no API/CLI

When NOT to use

• Prefer the app's own API or MCP server, connectors, shell, filesystem, or browser automation (Playwright) first

Mandatory first steps

If the app ID is unknown, first use discover_applications({query, include_capabilities: true}). Use its targetApp when present; do not assume a desktop registration ID is a process target.

1. get_tool_guide({ task_description }) — pick scripting vs accessibility vs OCR vs coordinates
2. get_app_capabilities({ bundle_id }) — is the app scriptable? accessibility: { nodes, hasControls } says whether its tree has real controls (accessible means hasControls); a handful of nodes means it draws its own UI
3. Route order: run_script → accessibility (fill_form, click_element) → OCR (read_window_text, click_text) → coordinates (click, type, key) last

Route for apps that draw their own UI (macOS)

Unreal, Blender, games and Electron canvases expose an empty accessibility tree. Do not call get_ui_tree, find_element or list_menu_bar on them.

1. list_windows (kinds: main, document, dialog, panel, toast) to pick the window
2. read_window_text with a region (keeps it to a few dozen tokens): lines with box in window points; screen point = screen_origin + box. On-device, works on covered windows, activates nothing
3. click_text to press a label by what it says (match: exact | contains | regex, nth); wait_for_text (appears, or gone: true) or wait_for_stable after the click instead of sleeping
4. screenshot / zoom only when text is not enough (icons, 3D viewports); mouse_drag for orbit/pan These OCR tools exist on macOS only; on Windows and Linux they are not listed.

Hard rules

• Always set target_app (bundle ID macOS / process name Windows) or target_window_id — or call set_target once and omit it afterwards (get_target shows it; an explicit target on a call wins)
• One click {coordinate, button: left|right|middle, count: 1|2|3}; left_click, double_click and the rest are aliases of it
• Do not screenshot every step: use get_ui_tree / find_element where the app has accessibility, read_window_text where it does not; screenshot with a target captures that window (full_screen: true for the whole screen)
• Wait with wait_for_window, wait_for_text, wait_for_stable, not wait plus a screenshot loop
• Long text: write_clipboard + paste shortcut, not type
• On focus failure, follow suggestedRecovery (activate_window, unhide_app, open_application)
• Use focus_strategy: "prepare_display" only after a focus race