computer-use

Computer Use

Use the Controller-owned CLI for desktop work:

vibe computer run ./inspect-mail.mjs --arg mailbox=Inbox --arg query=@/tmp/query.txt
vibe computer observe apps
vibe computer observe windows
vibe computer observe window --arg pid=123 --arg window_id=456 --output /tmp/window.png
vibe computer zoom --arg pid=123 --arg window_id=456 --arg x1=0 --arg y1=0 --arg x2=800 --arg y2=600 --output /tmp/zoom.png
vibe computer click --arg pid=123 --arg window_id=456 --arg x=120 --arg y=80
vibe computer type --arg pid=123 --arg window_id=456 --arg text="hello"
vibe computer key --arg key=return --arg pid=123 --arg window_id=456
vibe computer hotkey --arg keys='["cmd","c"]' --arg pid=123 --arg window_id=456
vibe computer scroll --arg direction=down --arg pid=123 --arg window_id=456
vibe computer drag --arg from_x=10 --arg from_y=10 --arg to_x=100 --arg to_y=100 --arg pid=123 --arg window_id=456
vibe computer set-value --arg pid=123 --arg value=enabled --arg element_token=el_1
vibe computer clipboard read --arg include_text=true
vibe computer clipboard write --arg text="copied text"
vibe computer launch --arg bundle_id=com.example.App
vibe computer force-quit --arg pid=123

For multi-step work, write a normal ESM file and import the shipped SDK:

import { computer } from "@avibe/computer";

export default async ({ mailbox, query }) => {
  const mail = await computer.app({ bundleId: "com.apple.mail" });
  const window = await mail.waitFor({ role: "AXWindow", labelContains: mailbox });
  const search = window.element({ role: "textbox", labelContains: "Search" });
  await search.fill(query);
  await window
    .expect({ role: "textbox", labelContains: "Search" })
    .toHaveValue(query);
  return { mailbox, query };
};

Run it with vibe computer run ./inspect-mail.mjs --arg mailbox=Inbox --arg query=@/tmp/query.txt. Ordinary argument values remain strings; @file loads the file as UTF-8 text. The default-exported function receives one argument object. Its return value becomes the JSON result; an omitted return is reported as null, and any other return must be JSON-serializable.

The runner uses Avibe's shipped Node runtime and injects @avibe/computer; never install or resolve another package with that name. The SDK keeps element tokens private, re-observes once when the Controller explicitly reports a stale observation, and stops on ambiguous action outcomes. App and window handles are immutable process descriptors; after an app relaunches, acquire fresh handles with computer.app(...) before continuing.

Scripts that send, submit, pay, delete, or perform another irreversible action must follow verify -> confirm -> re-verify -> act. If the owner-confirmation primitive is unavailable, stop before the irreversible action and use an interactive flow. Save a proven script as a reusable Skill only after the owner explicitly approves saving it.

Every single-operation leaf accepts --args-json OBJECT, --args-json @FILE, repeatable --arg NAME=VALUE, and --json. The two screenshot operations also accept --output PATH; the CLI writes or confirms the caller-selected PNG and prints the resulting path. observe window and zoom are the observation operations that establish the exact window state required before input.

--json keeps the structured driver object at result.structuredContent. The legacy content text block remains available for existing callers.

During one Agent Turn, consecutive vibe computer calls in the same Avibe Session share the driver session. Element tokens from one observation stay valid for following calls until a newer observation replaces that snapshot, so a short script can observe once, click several times, and observe once at the end:

vibe computer observe window --arg pid=123 --arg window_id=456
vibe computer click --arg pid=123 --arg window_id=456 --arg element_token=el_1
vibe computer click --arg pid=123 --arg window_id=456 --arg element_token=el_2
vibe computer observe window --arg pid=123 --arg window_id=456