computer-use

Computer Use

Computer Use lets Claude control a virtual computer — take screenshots, move cursor, click, type. Use it when there's no API, not as a first resort.

When to Use

• Automating legacy apps with no API
• Cross-app workflows (copy from A, paste to B, click approve)
• QA/UI testing where you want the agent to drive a real browser
• End-to-end automation mixing web and desktop

When NOT to Use

• Anything that has an API — APIs are 10-100× faster and cheaper
• High-frequency tasks — each action is a model call (slow)
• Anything where a failed click has real consequences (payments, deletes) without human in the loop
• Untrusted environments — the agent can be injected via screen content

Tool Shape

const tools = [
  { type: "computer_20250124", name: "computer", display_width_px: 1280, display_height_px: 800, display_number: 1 },
  { type: "text_editor_20250124", name: "str_replace_editor" }, // optional
  { type: "bash_20250124", name: "bash" }, // optional
];

const response = await client.beta.messages.create(
  {
    model: "claude-sonnet-4-6",
    max_tokens: 4096,
    tools,
    messages: [{ role: "user", content: "Open the browser and find Claude's API pricing page." }],
  },
  { headers: { "anthropic-beta": "computer-use-2025-01-24" } },
);

The model returns tool_use blocks with actions like screenshot, left_click, type, key, mouse_move, scroll.

Reference Implementation

Anthropic ships a reference Docker container (ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest) with:

• Virtual X server (Xvfb)
• Firefox + common apps
• VNC for human observation
• HTTP shim that executes tool calls

Don't run this on a host with real user data. It's a demo. For production, use a sandboxed cloud VM (Firecracker, Vercel Sandbox, cloud-hypervisor).

Action Loop

async function run(goal: string) {
  const messages = [{ role: "user", content: goal }];
  while (true) {
    const response = await client.beta.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 4096,
      tools,
      messages,
    }, { headers: { "anthropic-beta": "computer-use-2025-01-24" } });

    messages.push({ role: "assistant", content: response.content });
    if (response.stop_reason !== "tool_use") return response;

    const results = [];
    for (const block of response.content) {
      if (block.type === "tool_use") {
        const output = await executeAction(block.name, block.input); // your VM controller
        results.push({
          type: "tool_result",
          tool_use_id: block.id,
          content: output.screenshot ? [{ type: "image", source: output.screenshot }] : output.text,
        });
      }
    }
    messages.push({ role: "user", content: results });
  }
}