planchette AGENTS.md

Driving a window with the planchette CLI

You (the agent) are the control loop: capture the window, look at the screenshot, decide one action, run it, look again.

Setup

• planchette must be on PATH (uv tool install . from this repo).
• macOS: the terminal app needs Screen Recording + Accessibility permission (System Settings → Privacy & Security), else captures are black and input does nothing.
• Windows: no special permission. Linux: X11 only (Wayland unsupported).

Selecting a window

• Interactive (macOS): planchette pick — blocks until the user hovers and clicks a window under the overlay; allow a generous timeout (120s).
• Headless / any OS: planchette pick --list to see candidates, then planchette pick --name "Telegram" (case-insensitive substring of app name or title; frontmost match wins).
• planchette agent is the human-facing hotkey panel (it spawns its own agent session) — as an agent, never run it; use pick.

The loop

1. planchette capture → prints PATH WIDTHxHEIGHT. View that PNG. All coordinates below are pixels in this image, origin top-left.
2. One action:
   • planchette click X Y (--double for double-click, --right for context menu)
   • planchette drag X1 Y1 X2 Y2 (select text, move sliders, drag & drop)
   • planchette move X Y
   • planchette type "text" (types into the selected window — click the field first)
   • planchette key "cmd+t" (cmd/ctrl/alt/shift + key; --repeat N to press N times)
   • planchette scroll X Y up|down [lines]
3. Re-capture and look again. Repeat until done.

Text too small to read? planchette capture --crop X Y W H saves a native-res zoom of that region (coords in last-capture pixels). It's a magnifier only — keep computing click coordinates in the full capture's space.

Rules

• On macOS input goes to the selected window's process in the background — the user keeps their focus and cursor and can keep chatting while you drive. On Windows/Linux input follows real focus: warn the user not to touch anything.
• Never act blind: one action per capture, always look before the next.
• If actions have no visible effect in the next capture, planchette raise the window and retry once.
• error: no window selected / no capture yet → run the missing precondition and retry. window is gone → re-run pick.
• If the window falls behind another, planchette raise.
• Stop and ask rather than guess or do anything destructive.