hex AGENTS.md

HEX Agent Guide

Purpose

Build HEX as a local, observable macOS voice appliance with explicit Linux X11 and wlroots-compatible Wayland beta contracts. Keep the engine native Rust and keep consequential behavior explicit. Protected commands and typed captures remain compiled Rust; ordinary literal commands live in the explicit TypeScript user config. User-facing runtime settings persist in Application Support.

The distributed release starts in hotkey-dictation-only mode. Voice commands and their catalog remain available as a persisted opt-in that defaults off. Voice Action also requires its own persisted opt-in, defaulting off even for existing settings without an explicit enabled flag. DEVELOPER_FEATURES_ENABLED keeps meetings and their UI/CLI surfaces available only in debug builds.

Architecture

• audio: cpal device enumeration and timestamped mono float PCM delivery, live selection, and bounded stream recovery.
• moonshine: the only Moonshine C adapter and the streaming recognizer.
• suppression: the macOS event tap, shortcut suppression, and the configurable dictation-hotkey state machine.
• keyboard: active-layout key resolution and balanced synthetic shortcuts.
• dictation: warm pre-roll, growable capture, and 16 kHz local-transcription resampling.
• dictation_audio: the authoritative microphone timeline, lazy stream lifecycle, recording owner, exact shortcut boundaries, recovery handoff, and disposable bounded command audio projection.
• recording_environment: serialized RAII ownership of idle-sleep prevention, output muting, and supported media-player pause/resume behavior.
• dictation_processor: context-selected, deadline-bounded OpenCode rewrite profiles with raw-transcript fallback. The macOS app discovers the opencode2 beta executable, links missing installs to https://v2.opencode.ai/, and uses opencode2 api get to discover or start its managed service. Generation uses CLI-managed discovery with the matching owner-only service registration and authenticated loopback HTTP through the system curl; request bodies and credentials use stdin, never argv or new temporary files.
• parakeet: the strict-Metal transcribe.cpp adapter plus bounded inference, processing, ordered output, paste, last-result, and meeting-delta workers.
• apple_speech: the Swift SpeechAnalyzer bridge with per-locale support checks, asset reservation, and batch transcription.
• transcription: runtime selection and transactional warm-model activation.
• transcription_models: the compiled model catalog, language recommendations, pinned artifact verification, and lazy installation.
• transcription_service: bounded host-audio admission, hostile WAV validation, normalization, cancellation, and warm inference ownership.
• local_api: authenticated loopback discovery or direct-child endpoint handoff, bounded HTTP parsing, model preparation progress, and raw transcription routes.
• sdk/typescript: Promise and Effect v4 host wrappers for direct-child lifecycle, authenticated model preparation, and host-audio transcription.
• developer_control: the typed debug-only command/reply protocol the local API uses to drive the running app from hex dev.
• paste: clipboard insertion, continuation joins, and generation-safe clipboard restoration.
• accessibility: bounded focused-window and selected-text capture through Accessibility without touching the clipboard.
• recognition: the semantic coordinator for Moonshine, command mode, hotkey and voice-delimited controls, workers, and observations. It does not own the authoritative microphone timeline.
• command_grammar: typed command patterns, captures, overlap detection, and command catalog metadata.
• commands: pure contextual resolution plus bounded asynchronous macOS action execution.
• personal_commands: the Bun-hosted @hex/commands TypeScript workspace: managed SDK provisioning, the watch-reloaded user command and transformation host, bounded invocation dispatch, and status snapshots.
• text_replacements: phrase-boundary-aware compilation of configured text replacements applied longest-match-first to transcripts.
• config: compiled commands, preferred input devices, and dictation profiles.
• context: native foreground application and window context capture plus browser context. Browser host is the domain concept; Brave AppleScript is only the first adapter.
• application_catalog: cached installed-application discovery with normalized bundle identity and rendered icons for the UI.
• app_settings: persisted settings and live runtime projection for commands, hotkeys, microphone and transcription selection, recording behavior, processing, sound volume, and Dock policy.
• login_item: the native SMAppService.mainAppService adapter. macOS owns registration state; it is deliberately not duplicated in settings.json.
• app_paths: the Application Support owner for runtime logs and shared state.
• onboarding: required permission health, selected dictation-model installation, the release startup gate, and opt-in command-model setup.
• sparkle: packaged-app-only Sparkle lifecycle and manual update checks.
• linux, linux_app, linux_dictation, linux_input, linux_paste, linux_settings, linux_transcriber: the Linux beta CLI, GPUI shell, hotkey capture-transcribe-paste loop, persisted settings, and transcribe.cpp.
• linux_session: display-backend selection matching GPUI's nonempty WAYLAND_DISPLAY rule, independent of persisted preferences.
• linux_wayland_input: read-only evdev input, explicit physical key mappings, cancellable shortcut capture, exact chord state, and bounded device rediscovery.
• linux_desktop: the single process-owned GTK thread for the X11 tray and focus-free, click-through Wayland recording/processing HUD. A listener never initializes or shuts down a separate GTK runtime.
• linux_updater: signed direct-install updates, bounded downloads, atomic version activation, and restart handoff for user-local Linux installs.
• history: the owner-only bounded retained-dictation store: retention windows with hard entry and byte caps, atomic crash-safe persistence, and search. Text and bounded metadata only, never audio.
• events: bounded asynchronous append-only NDJSON observations and bounded incremental reading; dashboard and the GPUI Activity pane are read-only projections.
• desktop_activity: the shared listener, device, transcript, and session projection over EventReader.
• desktop_host: semantic desktop capabilities, portable UI snapshots, and typed actions implemented by the macOS root and contained Linux adapter.
• desktop_ui: platform-neutral GPUI visual tokens and controls shared by both desktop roots, including the mandatory pane scaffold: pane_header / pane_header_with_action, pane_body, pane_content, the shared header_button action chip, and the single PANE_CONTENT_WIDTH and PANE_LIST_WIDTH layout constants.
• text_input: the shared GPUI single- and multi-line text input with editing, selection, clipboard, and input-method support.
• desktop_transcription_picker: the single GPUI language/model picker used by both desktop roots over portable model presentation and platform preparation callbacks.
• app_window: the production Settings, Modes with mode-owned processing, Voice Action, History, Replacements, and opt-in Commands shell plus developer-only Meetings, Activity, and HUD Lab panes.
• status_item: the persistent macOS menu-bar owner for Settings, Paste Last Dictation, update checks, and orderly application shutdown.
• dictation_indicator: the click-through Metal/GPUI capture and processing HUD.
• meeting: explicit ScreenCaptureKit capture, owner-only WAV and transcript artifacts, final local-model publication, and recovery.
• meeting_live: bounded dual-stream Moonshine draft transcription and byte-offset transcript tailing.
• microphone_activity: permission-light CoreAudio process input observation.
• meeting_detection: pure provider classification, debounce, and suppression.
• meeting_watcher: GPUI application lifecycle, recognition ownership, meeting offers, and explicit handoff into capture.
• instance: the exclusive Application Support lock that prevents concurrent command listeners.
• feedback: bounded volume-controlled mode, capture, cancellation, and failure tones.