computer-use-forms

Operating controls

Goal: put a value in the right field, and know that it landed.

Order of preference. Each step down is slower and less certain, so only take it when the one above cannot do the job:

1. Accessibility — fill_form, set_value, click_element. Native apps, and a browser's own chrome including its dialogs.
2. The page — a browser-automation server acting by selector, or browser_find for an element's exact screen coordinates. Anything inside a web page.
3. OCR (macOS) — read_window_text with a region, then click_text on a label. Apps that draw their own UI have an empty accessibility tree; OCR needs no coordinates and works on a covered window. wait_for_text confirms a value or a message appeared.
4. Pixels — screenshot, then click and type. A canvas, a custom-drawn control, an app that exposes nothing, text OCR cannot read.

Facts you cannot infer, each measured

A browser exposes its own controls, never its pages. Measured on Chrome showing a web app: 37 accessible nodes, one text field — the address bar. A page input returns [] rather than an error, so two empty searches means stop searching, not search harder. Toolbar buttons and dialogs like "Save password?" are accessible.

A capture is smaller than the screen. 2560 wide arrives 1024 wide, so one image pixel is 2.5 real ones, against a field about 35 pixels tall. Every screenshot reply states its own mapping — read it rather than deriving one. On macOS 14+ a window capture is the window alone, without its shadow, so the mapping is exact; on older systems and other platforms a window capture can include a shadow and its scale is approximate. A window that could not be captured silently becomes a screen capture, and the reply says so. With a session target set, screenshot captures that window; full_screen: true captures the screen.

zoom returns 1:1. Without a target it takes desktop coordinates, so raise the window first or you will capture whatever else is at those coordinates. With target_app or target_window_id (since 7.5) the region is in that window's points, from its top-left.