deploy-runbook

Deploy runbook (API/frontend)

Remember stop-and-ask gate 2: production deploys need owner approval unless the owner's current instruction is itself the approval. Full runbook incl. LB/DNS/TLS/env/rollback/recovery: docs/ops/gcp-production.md. GPU workers deploy from their own workers/<name>/cloudbuild.yaml.

1. Reclaim disk first: npm run clean:worktrees (add --apply to act). Deploy worktrees are created by every deploy and removed by nobody. On 2026-08-04 ten had accumulated and filled the 126 GB disk to 100%, which fails in a way that reads like a different bug entirely: git worktree add dies mid-checkout with No space left on device and leaves a half-written tree behind. The script only reclaims a linked worktree that is detached, clean, and idle past --min-age-hours (default 2, so a concurrent agent's live build is never deleted); anything with uncommitted work is reported and kept. It applies the same rules to the subagent worktrees under .claude/worktrees/, which additionally must have their worktree-agent-* branch already merged into HEAD; the merged branch is deleted with git branch -d (on 2026-10-08 nine of those held 31 GB).

2. Prepare a CLEAN worktree: npm run prep:worktree -- --apply. That is the whole step; it encodes everything below, and running it beats staging by hand. Plan first with a bare npm run prep:worktree (writes nothing), --path <dir> to stage your own tree instead of colliding with a concurrent agent on the default /workspaces/.deploy-wt, --force to replace an existing tree (refused if it holds uncommitted work). What it does: git worktree add --detach <path> HEAD, then hardlinks every artifact in the ARTIFACTS list of scripts/prepare-deploy-worktree.mjs with cp -al on the same filesystem (not just the first). Three of them are build-critical, node_modules, chat/node_modules and character-studio/build; the optional ones (walk-sdk/dist, multiplayer/node_modules, animation-sources) are what let the worktree also run the repo's own test suite, so a pre-ship npm test can be taken at the exact commit being shipped instead of in the shared tree. It then copies BOTH .env and .env.local. Omitting either nested one kills the build minutes in: an OOM exit 144 on the avatar-studio rebuild, or Cannot find package '@sveltejs/vite-plugin-svelte' on the chat build. A nested artifact that is missing from the source tree is BUILT first (deps:chat, build:avatar-studio) rather than skipped, which is the failure a hand-run cp -al produces on a machine that has never deployed. .env.local is the one that carries DATABASE_URL, and without it step 3's migration gate cannot read the schema state and refuses to submit. Remove it when the deploy lands (git worktree remove --force <path>); that is what step 0 exists to clean up after. See docs/ops/gcp-production.md.

3. Build: npm run build:gcp. ORDER IS LOAD-BEARING and build:gcp already encodes it. Never hand-run the steps in a different order. The full chain, in order:

   check:conflicts → check:browser-graph → check:tdz-bootstrap → ensure:avatar-studio → build:info:snapshot → docs:freshness → build:lib:full (the UMD lib) → build:avatar-sdk → build:chat → build (the frontend vite build) → publish:lib → build:info → check:dist → check:pages

   Why the order is load-bearing: check:conflicts runs first so an unresolved merge-conflict marker fails in seconds instead of after a full build. check:tdz-bootstrap sits beside it for the same reason: it refuses a browser module whose top-level bootstrap call runs above the lets that call writes. JavaScriptCore checks an assignment target's temporal dead zone before evaluating the right-hand side and V8 does not, so that ordering ships green on Chrome and throws "Cannot access uninitialized variable." on every Safari, which is how /avatars/:id went dead on iOS and macOS while every desktop check passed. build:lib:full writes dist-lib/agent-3d.js, build:avatar-sdk copies that bundle into avatar-sdk/dist/index.mjs, and the frontend vite build resolves avatar-sdk/src/agent.js against it, so both must come BEFORE build; skipping them shipped whatever stale avatar-sdk/dist the build machine happened to hold, and in a fresh worktree failed the build outright with Could not resolve "../dist/index.mjs" (2026-08-17). The frontend vite build runs with emptyOutDir, so it WIPES dist/ and everything that writes into dist/ must come after it: that is why publish:lib, which mirrors the UMD lib into dist/, follows build rather than sitting next to build:lib:full, and check:dist fails without that bundle present. publish:lib writes only the moving channels (latest, 1, 1.5) from the fresh build; every released version directory is downloaded from the write-once archive gs://three-ws-lib-releases and checked against data/agent-3d-releases.json, so the build needs network access to storage.googleapis.com, and it fails if package.json names a version that was never cut. A versioned URL is immutable and SRI-pinned by embedders, so it must never change bytes: cut a new version with npm run release:lib after bumping package.json, never by rebuilding an old one (2026-10-01: rebuilding 1.5.2 on every deploy broke every published SRI pin). check:dist also fails when a documented SRI pin disagrees with the ledger; npm run check:sri-pins -- --fix rewrites stale pins. ensure:avatar-studio consumes character-studio/build and build:chat consumes chat/node_modules, which is exactly why step 1 stages worktree artifacts. docs:freshness measures documentation drift against git history and writes public/docs-freshness.json plus its summary, so it must run before build copies public/ into dist/; running it here is what keeps /docs/freshness and every docs-page freshness badge describing the commit actually deployed instead of whenever someone last ran it by hand. Do NOT hand-run build:vercel as the frontend build: it builds the SDK/lib sub-artifacts, NOT the static HTML pages, so dist/ ends up without /, /create, etc. and check:dist fails.

4. Submit: npm run deploy:gcp:submit. That is db:check (exit 4 if any migration is pending), then check:gcloudignore, then audit:deploy (no committed symlinks, no unsatisfied peer deps, no undeclared api/ imports, decoder assets present, and an installed node_modules that matches package-lock.json), then check:api-imports (imports every module under api/ and fails on any that throws while loading, since the server imports a handler on first hit and a load-time throw is a 500 on every request to that route rather than a build error), then the same gcloud builds submit --config server/cloudbuild.yaml --region us-central1 --project aerial-vehicle-466722-p5. check:gcloudignore simulates the upload and fails when a file the server imports at runtime would be excluded from the build context: an omitted services/home-relay/src/token.js shipped revision 00412 answering /api/healthz with ERR_MODULE_NOT_FOUND on every request (2026-09-04). The lockfile match exists because the image installs from the lockfile while every local test, and every deploy worktree that hardlinks this node_modules, runs whatever is on disk: a dependency bump nobody reinstalled shipped @x402/extensions 2.25.0 untested and every paid x402 route answered 500 on import (2026-09-17). If it fails, run npm install, then npm test, before deploying. Do NOT hand-run the bare gcloud builds submit: it is the only step that would have caught new code shipping over an old schema, and skipping it is how production ended up serving handlers that query columns their migration never applied (column "blocks_behind" does not exist, live on 2026-08-14). If the gate says migrations are pending, apply them first (npm run db:status, then npm run db:migrate); if it says DATABASE_URL is not set, step 1 did not copy .env.local. EVERY cloudbuild config must pin serviceAccount: .../three-ws-build@... (the default compute SA was deleted), and manual submits need --substitutions=SHORT_SHA=manual$(date +%s) when the config tags images with $SHORT_SHA.

5. Sync new crons, then purge CDN: npm run deploy:gcp:sync-crons, then npm run deploy:gcp:purge-cdn. The sync creates a Cloud Scheduler job for every cron in vercel.json that has none and touches no existing job; without it a new cron never fires (four sat dead for up to three weeks until 2026-10-08). The purge is synchronous on purpose. Do not re-add --async, or post-deploy checks read stale edge content and report phantom failures.

6. Verify the deploy landed. curl -s https://three.ws/api/version returns the live commit SHA + Cloud Run revision, and npm run smoke:prod sweeps every page declared in data/pages.json against the live site (deploy:gcp runs it automatically after the purge).

7. Check the workers too: npm run check:worker-drift. deploy:gcp:full ships three-ws-api only. Every service under workers/ deploys from its own cloudbuild.yaml and nothing else rebuilds it, so a merged worker fix can sit unbuilt for months (remesh-service segfaulted in production on a July image while its fix waited from 2026-08-11 to 2026-11). The check is read-only and needs a gcloud login: it compares each serving image's build time and commit tag with the git history of its workers/<dir>, prints DRIFTED/drifting/current/NOT-DEPLOYED per service and region, exits 1 when any worker has carried unbuilt source past the grace window (--grace-days, default 7), and prints the exact gcloud builds submit command per drifted worker. Each of those rebuilds is a production rollout under gate 2; the (extra) region copies it lists are config-only gcloud run services update --image rolls, which are pre-approved. It shells out to gcloud and git, so it runs on the operator machine before or after a deploy, not as a Cloud Run cron.