computer-use-and-browser-agents

Computer Use and Browser Agents

Purpose: Drive a browser or desktop reliably from an LLM, without leaking credentials, taking unsafe actions, or stalling on unfamiliar UI.

---

When to Use This Skill

• Automating workflows in apps that have no API
• Web scraping behind login, multi-step forms, captchas (where allowed)
• QA / test automation by LLM
• Internal RPA replacements
• Research agents that browse the open web

When NOT to Use

• A stable API exists -> use it; computer-use is slow, costly, brittle
• High-frequency / sub-second tasks -> too slow
• Strict legal / ToS constraints on automation

---

Approach Spectrum

[Screenshot + LLM]         <----- visual grounding ----->        [DOM + LLM]
  Anthropic Computer Use                                            browser-use
  OpenAI CUA / Operator                                             Playwright + LLM
  Most desktop agents                                               Most web-only agents

Visual approach: works on any UI but expensive (vision tokens, latency). DOM approach: cheaper and more reliable on web; fails on canvas/PDF/video.

Hybrid is common: DOM where possible, screenshot fallback for non-DOM regions.

---

Tool Surface

Typical action set:

Action	Args
screenshot	(returns image)
click	x, y or selector
type	text
key	key combo (Ctrl+L, Tab, Enter)
scroll	direction, amount
navigate	url
wait	ms or until-condition
read_page	(returns DOM-text or accessibility tree)

Anthropic Computer Use exposes computer, text_editor, bash. OpenAI CUA exposes computer. browser-use exposes a richer DOM-aware set.

---

Sandboxing (Required)

Never run computer-use against your own session, machine, or production browser profile.

Sandbox	Use
E2B, Browserbase, Scrapybara, Hyperbrowser	Managed sandboxed browsers/desktops
Docker container with Chrome + Xvfb	Self-hosted, ephemeral
Dedicated VM	Strongest isolation
Ephemeral browser profile	Per-task, deleted after