oxrsys AGENTS.md

OXRSys Runtime

OpenXR runtime that started on macOS and is now being moved toward a measured cross-platform runtime shape. The project currently combines the shared runtime, a unified macOS/iOS viewer with Simulator and StereoView modes, a first-pass visionOS viewer, a Quest/Pico-oriented Android VR client, and Linux-first Qt frontends. The repository also includes a native SwiftUI macOS Home app and a Qt Home app for compatible app launching, runtime selection, runtime configuration, and runtime registration workflows.

Current state: Metal/core runtime, Vulkan interop, Linux Vulkan/FFmpeg scaffolding, typed internal graphics/frame plumbing, release-time Metal streaming snapshots, portable platform/socket helpers, controller and hand input paths, loader-backed runtime tests, XR_EXT_conformance_automation, XR_EXT_hand_interaction, and XR_EXT_debug_utils are in place. Windows is scaffolded in layout/docs only for this pass. The Android VR client now feeds real XR_EXT_hand_tracking joints into the runtime, gates controller poses with explicit active flags, keeps hand-interaction bindings available alongside active controllers with controller-first priority, supports WiFi UDP and USB ADB reverse TCP streaming, matches per-frame render poses for headset compositor reprojection, reuses short decode/network gaps through configurable client reprojection with conservative optional pose warp, shows a local headset shell with status text, reset, controller lasers, hand laser/pinch input, visible hand-joint markers, a simple 3D grid, and optional XR_FB_passthrough while waiting for video, requests the server-selected display refresh rate, enables preset-driven dynamic XR_FB_foveation when the headset supports it, and supports the Quest shader path for foveated-encoding decompression plus edge-aware upscaling. ABR full mode can reconfigure encoded stream resolution over USB TCP through protocol v1.2 StreamConfigUpdate/Ack when the client supports it, without resizing the OpenXR application's swapchains, and the passthrough feature can keep Quest passthrough active during negotiated MR streaming while local shell GL resources are released. Runtime video sends now run behind a bounded encoded-frame sender queue so socket backpressure does not run inside VideoToolbox callbacks, the Quest client drains MediaCodec output off the XR frame loop, and the runtime ABR controller uses client latency, displayed frame age, keyframe requests, send/encoder drops, and reprojection pressure to adjust bitrate with sliding windows and hysteresis. The visionOS viewer now starts from a minimal floating search window, enters immersive VR automatically when the stream connects, and sends head pose, hand joints, and first-pass tracked accessory controller data while the immersive space is open. The macOS SwiftUI Home app now targets direct notarized distribution so it can scan known apps, launch compatible apps with the user-selected XR_RUNTIME_JSON, register that selected runtime, and capture app logs. The Home app shows a main-window runtime activity summary from ~/Library/Application Support/OXRSys/runtime_status.json, including idle/streaming state, transport, connected device family, active OpenXR application, WiFi/USB transport readiness, first-launch runtime registration guidance, one-step USB reverse setup, SDK-free native USB ADB setup in the SwiftUI Home app, native ADB-server protocol support with external adb fallback, and per-app custom ADB path selection for USB setup. Home streaming controls include the shared runtime 1-200 Mbps bitrate bounds, server-selected refresh rate, encoder preset, foveated encoding preset, ABR/dynamic-resolution mode, mixed reality, occlusion, spatial toggles, and a separate Headset Client section for client foveation override, Quest shader upscaling, client reprojection, and reserved headset-audio configuration; clients can send ClientConnect.maxBitrateMbps = 0 to use the server-configured bitrate without adding a client-side cap. The Home app can enable a Developer tab from its Settings tab, open the macOS simulator in a same-process window backed by the shared OXRSysSimulator Swift package, and show live runtime streaming statistics from the existing telemetry path. The Qt Home Developer tab opens the shared Qt simulator widget in a dedicated window with UDP video preview, mouse-driven synthetic head tracking, simulator-owned vertical FOV sent through tracking eye-FOV metadata, explicit FFmpeg-disabled fallback, frame-loss/FEC status, and keyframe recovery requests. The macOS package helper builds the runtime dylib and Home app into one local folder with a complete runtime/ directory; the distribution helper signs that package, creates a combined archive, and can submit that archive for notarization with Apple Developer account credentials. The net.demonixis.oxrsys-unity Unity Package Manager package under scripts/unity/ covers editor runtime selection and the macOS Player OpenXR loader bundle-name workaround needed by exported Unity apps. As of March 17, 2026, the pinned non-interactive OpenXR-CTS baseline is fully green locally: 63 passed, 36 skipped, 0 failed.