computer-use

Computer Use — Verified-Aim Desktop Automation

The agent sees and controls the desktop through a closed feedback loop designed for surgical accuracy with any vision-capable model:

1. One coordinate space, owned by the plugin. Every image the tools return (screenshot, zoom, or click magnifier) becomes the current frame. The model always gives coordinates as pixels read off the latest image; the plugin does all translation to real screen position — screenshot downscaling, Retina/HiDPI scale factors, and multi-monitor offsets. The model never does coordinate math, which removes the entire class of "right target, wrong space" misses.
2. A crosshair marks the cursor on every returned image, so the model always knows where the pointer actually is. In aiming frames (zooms and magnifiers) thin hairlines additionally run from the image edges through the exact cursor pixel — a fat ring can visually swallow a 16px control that is actually 15px away, while a hairline either passes through the target or visibly does not.
3. Zoom for small targets. computer_zoom re-captures a chosen region at native resolution (up to 4x magnification). The zoomed image becomes the frame, so tiny controls are clicked in a space where they are dozens of pixels wide. A zoom below 2x is flagged in the result, with the region size that would reach 3x — weak "zooms" of wide regions are where small-target clicks historically missed.
4. Every click returns proof — including an objective change report. computer_mouse_click captures the screen before and after the press and reports what percentage of pixels changed around the cursor and across the display. "NO visible change" after a click that should have had an immediate local effect (closed a tab, opened a menu) means the click missed, regardless of how the close-up reads. The check samples one display for under half a second, so slow-painting results, cross-display effects, and sub-threshold changes evade it — the model is told to verify with a screenshot before retrying, and never to blindly re-click side-effectful controls. Click, move, and drag also return a 3x-magnified close-up with the crosshair and hairlines on the exact pixel acted on, which becomes the current frame so corrections happen in finer coordinates instead of by re-guessing on the full screenshot.

This mirrors the practices Anthropic uses for Claude's own computer use: act on the latest image only, verify after every action, zoom rather than squint, prefer keyboard shortcuts when they are more reliable than pointing.