control-window

Controlling a macOS window

You drive a real macOS window through the planchette CLI. You ARE the agent loop: capture the window, look at it, decide the next action, run it, look again.

Prerequisites

• planchette must be on PATH (uv tool install . from the plugin repo).
• The terminal app running your agent needs Screen Recording and Accessibility permission (System Settings → Privacy & Security). Without them, captures come out black and input does nothing.

The loop

1. Pick the target window. Run planchette pick. A dim overlay appears — tell the user to hover the window they want and click it (Escape cancels). The command blocks until they click, so allow a generous timeout (e.g. 120s). On success it selects and raises that window. If the user already named the app, skip the overlay: planchette pick --name "Telegram" selects the frontmost match by app/title substring, and planchette pick --list shows the candidates.

2. Capture. Run planchette capture. It prints PATH WIDTHxHEIGHT. View that PNG with your file-reading/image tool. Every coordinate you pass below is a pixel in this image: (0,0) at top-left, up to WIDTH×HEIGHT.

3. Act — one action, then look again. Pick the single next step toward the goal and run the matching command:

   • planchette click X Y — left click (--double for double-click, --right for a context menu)
   • planchette drag X1 Y1 X2 Y2 — press-drag-release: select text, move sliders, drag & drop
   • planchette move X Y
   • planchette type "text" — types into the selected window; click the field first
   • planchette key "cmd+t" — combos use cmd/ctrl/alt/shift + a key; add --repeat N to press it N times (e.g. arrow-key navigation)
   • planchette scroll X Y up|down [lines]

   Text too small to read in the capture? planchette capture --crop X Y W H saves a native-resolution zoom of that region (coords in last-capture pixels). It's a magnifier only — keep computing click coordinates in the full capture's space.