computer-use

Computer Use (Mac via Autonomous Buddy)

Quick Start

Autonomous Buddy is a small macOS app the user installs on their Mac. Once paired with the device via the Buddy card on the Monitor web UI, it lets the device launch apps, open URLs, type text, fire keyboard shortcuts, and click UI elements on the user's actual Mac.

This skill emits inline markers that the device fires asynchronously while TTS speaks the confirmation:

[HW:/buddy/exec/<action>:<params-json>]

The marker hits /api/buddy/exec/<action> on the device, which dispatches over the buddy's persistent WebSocket. Fire-and-forget — no response is awaited (TTS continues immediately).

Workflow

1. Determine the user's intent and pick one or more actions from the table below.
2. Build the marker(s) — flat params JSON only (no nested objects).
3. Place markers at the start of the reply, then add a short confirmation that TTS will speak.
4. If no Mac is paired, say so and tell the user to set it up via the device's web UI Buddy card.

When to load reference/vision.md instead

The marker pattern below covers ~90% of computer-use requests: launching apps, opening URLs, typing into the focused field, keyboard shortcuts, named-button clicks. It is fire-and-forget — fast, but cannot return data.

Load reference/vision.md and follow its synchronous bash/curl loop only when the task requires actually seeing the screen:

• "Click the blue button in the toolbar" / "click the X on that dialog" (no stable accessibility label)
• "What's on my screen right now?" / "Read me the error dialog"
• "Drag the slider to the middle" / "move that window over here"
• Multi-step UI navigation where each step depends on what appears next

Do NOT load vision for tasks the marker actions already handle — vision is slower and far less reliable (~22-40% per multi-step task).

Examples

Input: "Open Chrome on my computer" Output: [HW:/buddy/exec/open_app:{"app":"Google Chrome"}] Opening Chrome on your Mac.