gui

gui — cross-driver GUI actuation

Whenever the active driver runs with ui_mode=gui (or desktop), sim serve injects a gui object into your sim exec namespace alongside session / solver / meshing / model. The object is the same shape across solvers — only the process filter differs — so one skill serves every GUI-capable driver.

/connect advertises it:

{
  "ok": true,
  "data": {
    "...": "...",
    "tools": ["gui"],
    "tool_refs": {"gui": "sim-cli/_skills/sim-cli/gui/SKILL.md"}
  }
}

If tools doesn't contain "gui", the driver launched headless and the object is absent — don't call it.

When to reach for gui

Three scenarios dominate:

1. A blocking dialog is wedging the workflow. A login prompt, overwrite confirmation, or script-error dialog can pause agent work until someone clicks a button — that someone is you, via gui.
2. You need to drive the GUI where the SDK can't. Some workflows expose a UI-only surface that the driver API does not cover.
3. You need a per-window screenshot. sim screenshot captures the whole desktop. SimWindow.screenshot() captures just the window you care about — cheaper to read, less visual clutter for the LLM.

If the SDK has a programmatic path (session.tui.*, model.solve(), ModelUtil.loadCopy()), prefer that. gui is for the UI-only surface that the SDK doesn't cover.

Remote equivalence

gui is in the session namespace on the sim serve side. You talk to it via the existing /exec HTTP channel:

# local Windows box
sim exec "dlg = gui.find('Login'); dlg.click('OK')"

# Windows box on the LAN / Tailscale
sim --host 10.0.x.y exec "dlg = gui.find('Login'); dlg.click('OK')"

No new endpoint, no new protocol — the same API shape from anywhere the agent runs.

Requirement on the server host: sim serve must run in a real interactive desktop session (normal login or RDP). Windows service / SSH session 0 has no desktop, so pywinauto can't enumerate any windows even though the solver processes are running. This is the same constraint GUI-capable drivers document.