unix_computer_use

Unix Computer Use

This bundled extension exposes a reviewable computer-use substrate for local macOS/Linux desktops and explicitly configured remote targets. All actions are for supervised, low-risk workflows under human or benchmark observation. Windows support is deferred to a separate backend/skill.

Tools:

• capabilities reports the platform, display-session type, and detected screenshot/input backends plus the active connection/backend.
• list_connections, add_connection, test_connection, activate_connection, and use_local manage the local/remote connection registry in skill state.
• screenshot captures the desktop, downscales the image to fit WXGA (1280x800, configurable) and persists the exact image-to-input coordinate transform. Pass coordinates read off the returned image directly to the input tools — they are remapped automatically. The result carries the typed auto_attach_image field, so the host attaches the image to the conversation in the same round (v6.81.1) — no view_image call is needed for a fresh capture; the returned path stays view_image-readable for re-viewing later.
• click (left/right/middle, double/triple), the thin aliases double_click/triple_click (models call these names unprompted), move, left_click_drag, mouse_down/mouse_up, cursor_position, type_text, key, hold_key, scroll, and wait execute input through platform tools when available. Pointer coordinates pass through one normalizer that accepts the malformations models actually emit (the pair packed into x with y absent or duplicated) and fails loudly on anything ambiguous.
• window_list lists visible windows/processes where the platform exposes a lightweight backend.
• ax_tree returns a set-of-marks accessibility snapshot of the frontmost window on macOS: numbered interactive elements (role, title, center coordinates in input space — click them with raw=true). It degrades honestly to a process/window list when the AX walk fails or on Linux.
• remote_exec runs a shell command on the active remote backend only; it refuses on local backend. Use ordinary Ouroboros shell/file tools for local work.

Coordinate contract:

• Input tools accept coordinates in the LAST screenshot's image space and remap them through the stored transform; raw=true bypasses remapping.
• macOS input consumes LOGICAL points while screenshots are physical pixels; the stored transform already folds the Retina scale in. On multi-display Macs the scale is approximate (approx flag) — prefer ax_tree marks.

Backend matrix (honest limitations):

• macOS: screencapture + cliclick + osascript; sips for downscaling. scroll is unsupported (cliclick has no scroll-wheel command — page via key with page-down/page-up); middle click and non-left mouse_down/mouse_up are unsupported via cliclick. hold_key holds PURE-MODIFIER combos only (cmd, cmd+shift, ...): cliclick kd:/ku: accept only modifiers and kp: cannot hold — non-modifier holds report an honest error. In key combos the base key is tapped while modifiers are held (cmd+s = hold cmd, tap s). TCC permission state (Screen Recording / Accessibility) is NOT verified: tools exit 0 even when denied (wallpaper-only capture / dropped input) — ensure grants.
• Linux X11: xdotool (input), gnome-screenshot/scrot (capture), wmctrl (windows), ImageMagick (downscaling). Function keys and case-sensitive keysyms are supported (f5→F5, XF86AudioPlay as-is).
• Linux Wayland: ydotool (pointer + typing; requires a running ydotoold) or wtype (typing only), grim or gnome-screenshot (capture). xdotool does NOT work on Wayland. key and hold_key are unsupported on Wayland (ydotool key takes raw keycodes only — combos would silently no-op, which this skill refuses to fake); use type_text for text. Pointer press/release uses ydotool button masks (0x40/0x80).
• Remote OSWorld HTTP: screenshot uses GET <target>/screenshot; input and remote_exec use POST <target>/execute, matching OSWorld's in-VM pyautogui channel. The runner writes an osworld_http connection before each task. Honesty guarantees on this backend: scroll clicks are 1:1 wheel detents (not multiplied); non-ASCII type_text is pasted via the in-VM clipboard (pyperclip + ctrl+v, ASCII typewrite fallback) so Unicode is not silently dropped; every action's ok reflects the guest process returncode/status (HTTP 200 alone is not success); key backspace presses BackSpace and fwd-delete presses forward Delete; screenshots are capped at 20 MB and PNG-validated before use.
• Remote SSH macOS: screenshot uses remote screencapture + scp; input uses a remote input helper path. The remote Mac must have Screen Recording and Accessibility grants configured by the owner. This skill does not install or store private SSH keys.