oxygen: Skill for Claude Code

Oxy multi-instance scaling — quick reference

The current architecture in one line: Oxy runs as a split fleet keyed by OXY_ROLE — a single stateful ide (the Factory: working copy + .git + compile + chat-run execution), a horizontally-scaled stateless serve fleet (reads Postgres + S3 only), and a worker queue drainer. This is a Postgres primary/read-replica shape: the ide is the primary, the serve fleet is the read replicas, and the compile boundary (compiled *_definitions rows + S3 blobs, keyed by revision_id) is the replicated read model.

The direction (decided 2026-10-05): the Factory is being retired, not scaled. No IDE; content is authored in git and pushed; compile becomes its own service that fetches a commit and serves previews and sandboxes; chat and every other surface read the compiled revision only; workspaces with no git remote go. Design new work so it needs no server-side working copy — a change that adds a reason to keep the Factory is going the wrong way. Plan, production data and open decisions: internal-docs/factory-retirement.md.

Read these first when grounding a decision:

• internal-docs/multi-instance-fleet.md — how the fleet works today (roles, the stateful-vs-HA matrix, route classification, the super_read_only guard, graceful degradation, code map). The primary reference.
• internal-docs/factory-retirement.md — the forward plan: phases, exit tests, and the existing statements it overturns.
• internal-docs/multi-instance-fleet.md — the original phase ledger + the rejected-alternatives list.
• internal-docs/compile-boundary.md — operator runbook.

What is built (the current reality)

• Split fleet via OXY_ROLE (ide | serve | worker | all). One knob selects the topology; everything else derives from it. A serve replica derives in-process workers + the periodic global driver OFF; every other role ON. The legacy flags (OXY_DISABLE_INPROCESS_WORKERS, OXY_INPROC_GLOBAL_WORKER, --no-workers) survive only as two-directional overrides.
• Compile boundary (compile-complete stateless serving). The ide compiles the working copy → Postgres *_definitions rows + S3 blobs per revision_id → promotes via workspaces.current_revision_id. The serve fleet reads the promoted revision and never walks the workspace FS. This is why the read path needs no working copy — and therefore no per-request clone.
• Self-routing. role_manifest::classify is the single routing authority; a serve replica reverse-proxies IdeOnly requests to OXY_IDE_UPSTREAM (ide_proxy). Replaces the drift-prone external route table that caused three outages.
• Route classification + HA carve-out. Every route states its pod at the mount — route_ide (FS/exec/live-stream) or route_fleet (Postgres/S3 reads stay HA) — and the two take differently-typed handlers, so a working-copy handler cannot compile onto a fleet route. Plus drift tests and the super_read_only runtime guard. See the oxy-route-classification skill.
• Runtime-artifact S3 mirror (runtime_artifact.rs) — charts/results/app-data mirror to the compile-boundary bucket so any replica serves them; ide-down charts degrade to the S3 mirror.
• Schedules/monitors fire without a leader. The periodic global driver runs on every eligible node; tick_schedules/tick_monitor_schedules CAS-advance next_run_at so firing is exactly-once across replicas. (Leader election was tried and removed — the CAS already guarantees it, and running on all eligible nodes is better HA.)
• Backpressure — admission control (global ceiling + per-tenant fairness, 503 + Retry-After), worker HPA on outstanding work (queued + claimed) against capacity — see the recipe in internal-docs/worker-fleet.md; queue depth alone goes absent when the fleet keeps up.
• Migrations — a dedicated migrate Job owns the schema; serve/compile/worker honour OXY_SKIP_MIGRATIONS; a startup advisory lock serialises co-booting nodes.
• Main traffic off the Factory (October 2026). Custom-app function calls run on any replica, with a handler-level replay to the Factory for a workspace with nothing compiled or a database that is a file in the checkout. Airway start, single-window backfill, cancel, live stream and both resets are served by any replica. Custom-app procedure runs execute on the task queue, at most once, with an execution heartbeat. OXY_IDE_DEFER_QUEUE_WORK (off by default) makes the Factory drive only compiles and take anything else after 30 s unclaimed.