macos-app-automation

macOS App Automation

Use this skill when the user asks whether Hermes can control a native macOS app, or asks you to verify, configure, or troubleshoot AppleScript/Automation access for an app.

Preferred automation ladder

1. AppleScript dictionary / app scripting API

• Best for structured app data and repeatable operations.
• Verify with the app's bundle ID, version, and scripting dictionary before promising coverage.

1. App URL scheme

• Good for quick-add or deep-link actions when documented by the app.

1. System Events / Accessibility scripting

• Useful for menu items and UI operations not exposed in the app dictionary.

1. Hermes computer_use

• Last-resort GUI operation for UI-only features. Capture first, click by element index, verify after state changes.
• If the wrapper reports that its cua-driver session ended, recover the same session before concluding GUI control is blocked. If Hermes has already rejected repeated identical calls, change the wrapper call signature or continue through direct cua-driver call operations rather than retrying the same arguments.
• Do not turn an internal wrapper guardrail into the user-facing result. Recover through the direct driver path first; if a real blocker remains, report it in task language with the evidence that distinguishes it (for example, asleep display versus locked login screen).
• On an unattended Mac, wake a black/asleep display non-destructively and recapture before diagnosing capture failure. A visible login/password screen is a credential gate: never type, request, infer, or bypass the password, and never claim a click reached an app behind the lock screen without verified state change.
• After the user unlocks the Mac for a long unattended native-app workflow, start a tracked /usr/bin/caffeinate -dimsu process immediately and stop it during cleanup; caffeinate -u -t 2 wakes only and does not prevent a later relock.
• Electron/webview drawers can retain off-screen controls in the accessibility tree with zero- or one-pixel bounds. Programmatic set_value by accessible label may still work, but never use an off-screen element as a human handoff target—especially for credentials. Scroll the exact row into view, recapture, require visible nonzero bounds, focus the intended field, and verify the adjacent action before asking the user to paste or confirm. After Save, recapture neighboring ordinary fields such as Endpoint and Model to catch a paste that landed in the wrong control without echoing its value.
• Treat a modal dismissal as coordinate-invalidating: perform one click, recapture, and only then choose the next target. Never batch raw coordinate clicks across a dialog close.
• See references/cua-session-recovery.md for deterministic session revival, black-capture diagnosis, lock-screen handling, repeated-call-guard fallback, and modal-safe clicking.