screenshotting-gui

Screenshotting a GUI app

Produce documentation-grade screenshots of a running GUI application without a window manager and without touching the operator's screen. The app runs on a throwaway Xvfb display; you drive it to the exact state you want with synthetic X input, capture the frame, and annotate it.

Everything the model already knows (what X11, ImageMagick or a clipboard are) is omitted on purpose - this file is only the parts that are easy to get wrong.

The one rule that keeps it safe

Run on display :99 (Xvfb), never :0. :0 is the operator's live desktop - clicking, typing and screenshotting it is a privacy and correctness disaster.

The subtle trap: xctl.py / xpaste.py default their own target to :99, but any GUI app you launch reads $DISPLAY. If $DISPLAY is still :0, the app opens on the real screen while your captures read :99 - winlist comes back empty and you capture a black frame. Always export DISPLAY=:99 for the app launch, not just XCTL_DISPLAY.

Toolchain

Deterministic pieces live in scripts/ (run any with --help):

Script	Role
scripts/xctl.py	Drive the app: move click key type scroll drag place shot winlist via python-Xlib XTEST.
scripts/xpaste.py	Serve UTF-8 on the CLIPBOARD so key ctrl+v pastes unicode / non-Latin text.
scripts/annotate.py	ImageMagick post-processing: box (rounded highlight), crop, vstack, arrow.

Requirements: Xvfb, python3 + python-Xlib, ImageMagick (import, convert). Capture default is ImageMagick import (no extra dependency, no max-size ceiling). Fallback if ImageMagick is unavailable: python-Xlib root.get_image + Pillow (~2x faster, but needs Pillow and must tile a root above ~4.2 MP; pixel order is BGRX, not RGB).

Workflow

DISP=:99
export XCTL_DISPLAY=$DISP
S=scripts   # path to this skill's scripts

# 1. Isolated display
Xvfb $DISP -screen 0 1920x1080x24 -ac >/tmp/xvfb.log 2>&1 &
sleep 1; DISPLAY=$DISP xdpyinfo >/dev/null || { echo "Xvfb down"; exit 1; }

# 2. Launch the app ON :99 (export DISPLAY, not only XCTL_DISPLAY)
DISPLAY=$DISP <app> &            # e.g. an IDE, a browser, a native app
sleep 2                          # heavy Electron/JVM apps need 3-8 s; poll with winlist
python3 $S/xctl.py winlist       # confirm the window is on :99, note its geometry

# 3. Drive it to the exact state you want (see command table below)
python3 $S/xctl.py click 1490 1003        # open a menu, toggle a panel, etc.
sleep 0.5

# 4. Capture (full, or crop to the region)
python3 $S/xctl.py shot /tmp/raw.png --crop 1270 560 650 520

# 5. Annotate and save
python3 $S/annotate.py box /tmp/raw.png out.png 1300 600 1720 680