vision-bridge-windows-automation

vision-bridge: Windows desktop automation

vision-bridge is one MCP server that gives the agent four verbs for any window on Windows: capture (see), act (do), find (locate), wait_for (wait). The screen comes back as structured text, so this works for text-only models too. Elements are addressed by a stable id - never by pixel coordinates or CSS selectors.

Repository: https://github.com/rusnetru/vision-bridge (MIT)

When to use

Use it when:

• the task lives in a Windows GUI program with no API or CLI (ERP/accounting clients, installers, settings dialogs, legacy WinForms/WPF apps, Notepad, Explorer);
• an Electron/Chromium desktop app must be read or clicked (OCR fallback handles it);
• a web task needs the user's own logged-in Chrome session.

Do NOT use it when:

• a CLI, API, file read or a dedicated MCP server can do the job - those are faster and exact;
• the machine is not Windows (UIA and OCR backends are Windows-only);
• the action is irreversible (payments, deletes, sending messages) and the user has not confirmed it in this conversation.

Setup check (first call)

1. Call capture("") (foreground window). If the tool is missing, vision-bridge is not connected - show the user reference/setup.md and stop.
2. If the result has ok: false and an error like backend 'uia' unavailable ... uv sync --extra desktop, relay that exact install command to the user. Do not try to install packages yourself unless asked.

Core loop

capture(target)            -> read `text`, pick the element from `elements[]`
act(id, action, ...)       -> click / type / set_value / focus / read / scroll / double_click
capture(target) again      -> verify the result before the next step

Rules:

• Re-capture after every state-changing action. Ids are valid for the latest capture/find; after a click the window may change and old ids go stale.
• Verify, don't assume. After type or set_value, call act(id, "read") and compare content with what you meant to enter.
• Prefer set_value over type for fields you want to replace entirely; use type for appending or for controls that react to keystrokes (search boxes, autocomplete).
• Use find for one known element, capture when you need to understand the screen.
• Use wait_for instead of sleeping after anything that loads: dialogs, installers, page navigation. Set timeout_s to the realistic wait (30-120 s for installers).