canvas

chrome-use canvas / WebGL / games

Canvas / WebGL UIs (game boards, voice-room mic seats, map tiles, design canvases). These paint to a <canvas> — there is no DOM node and no accessibility node behind what you see, so snapshot/find/eval querySelector will never return a ref for them. This is a hard limitation, not a missing feature. To work with them:

• Read the rendered pixels with chrome-use canvas list then chrome-use canvas capture [selector] <file> (extracts the canvas bitmap), or a normal screenshot of the region — then you interpret it.
• Act by coordinate: compute the target point and chrome-use click <x> <y> (or box @ref on a container to get its CSS-px box first). Coordinates are the correct tool here — the snapshot-first rule explicitly carves out canvas.
• On the relay, a coordinate click can drift onto the user's foreground tab; prefer a --launch/owned tab for heavy canvas coordinate work, or confirm the underlying state via the app's backend/API instead of driving the canvas.

Canvas / WebGL apps (games, map & 3D viewers, drawing tools)

These paint everything to a <canvas> and expose almost no accessibility tree, so snapshot comes back near-empty and refs are a dead end. snapshot detects this, prints a one-line hint, and also saves a viewport screenshot and prints screenshot: <path>. View that image instead of calling screenshot again (JSON: data.screenshot, screenshotReason: "sparse"; opt out with AGENT_BROWSER_SPARSE_SCREENSHOT=0). Drive them the screenshot way:

chrome-use canvas list                 # enumerate <canvas> elements (size, type)
chrome-use canvas capture out.png      # save the canvas's RENDERED pixels to PNG —
                                          # toDataURL (full backing-store res, e.g.
                                          # Figma 2522x1904), screenshot fallback for
                                          # WebGL w/o preserveDrawingBuffer / tainted.
                                          # Gets the RENDER, not hidden source data
                                          # (those live in the app's binary store/API).
chrome-use screenshot /tmp/s.png       # SEE the state (your only read path —
                                          # eval/get text return nothing useful)
chrome-use click 640 360               # interact by viewport coordinate
chrome-use press d --hold 800          # hold-to-move, precise (timed in-daemon —
                                          # NOT keydown+shell-sleep+keyup, which
                                          # adds ~250ms jitter per round-trip)
chrome-use press Space                 # discrete actions (jump/attack/confirm)