drive-browser

Drive browser with Playwright

Use Playwright for precise browser actions and for a screenshot-and-click loop. Pick how to interact by what the page exposes, not by habit.

Decide: two independent choices

How you find and act on elements:

• Locators (default). getByRole/getByLabel/getByText first: they auto-wait and re-resolve as the page changes. Use CSS only when a locator can't express the target. This handles your own app and most real sites.
• Vision: only when the structure is genuinely opaque (custom canvas, a widget locators can't reach): screenshot → read it → click coordinates.

Where the page runs:

• Clean state or another login: create a new browser context, either in an attached browser or in a fresh launch. Never clear the user's context to get a clean one.
• The user's existing login: attach to the real browser and use its existing context. Read Safety first; open a task-owned tab rather than taking over an existing one unless asked.

These choices are independent: locators work with an existing login, and vision works in a fresh context. See setup and lifecycle recipes for connection, launch, and cleanup. For multi-step exploration, keep the browser and page handles in the host's persistent Node REPL rather than reconnecting for each action.

Locator mode

Prefer page.getByRole('button', { name: 'Save' }) and getByLabel('Email'). Use page.locator(css) only when needed, and page.evaluate(fn) for DOM reads that locators don't express. Wait for the state you need (locator.waitFor(), page.waitForURL(), page.waitForFunction()), not a guessed delay. For CSS-media features, use page.emulateMedia({ reducedMotion: 'reduce' }).

Vision mode

Work one step at a time: screenshot → read the PNG → choose coordinates → page.mouse.click(x, y) → screenshot again. Don't write a blind script with guessed coordinates.

Take viewport screenshots with page.screenshot({ path: 'step.png', scale: 'css' }). Each image pixel then matches a CSS pixel, so use the image coordinates directly in page.mouse.click. Read the image at its original dimensions; a resized preview changes the coordinates. Re-screenshot after anything that moves controls.