house-trainer

house-trainer — the EP-T QLoRA pipeline playbook

Everything here was proven live on the trainer (T0–T3, LOG 2026-07-22..24). Follow it and you skip a week of Windows-ROCm debugging. Read the two absolutes first.

Absolutes

1. Privacy: job-application/CV content NEVER enters training data. Every build runs the leak gate (scripts/train/lib/privacy.mjs → scripts/longview/lib/leak_gate.mjs). Quarantined sids (live home + sessions workspace quarantine.json) are dropped structurally. Generated rows are git-ignored — never commit them. Never weaken scripts/train/dataset/personal_terms.json to make a build pass.
2. Honesty: rubric (scripts/train/eval/rubric.md) is frozen BEFORE training. If a run invalidates an earlier number, the correction goes on the record (docs/train/T3-eval-report.md v1→v2 precedent). Tuned-worse is a valid, logged result.

Liveness — NEVER trust a tqdm log for a long GPU job

The eGPU wedges transiently (TB3/RDNA4): a step hangs, the process stays alive but blocked, working set collapses to ~MB, and the tqdm log freezes looking exactly like "slow". A DPO run sat wedged at step 8/38 for 6 h this way. Prove liveness, don't infer it:

• log mtime vs now (stat -c %y <log>) — hours stale on a minutes-long job = wedged;
• two CPU-time snapshots ~90 s apart (Get-CimInstance Win32_Process ... UserModeTime+KernelModeTime) — a live job burns CPU seconds, a wedged one doesn't;
• advancing artifacts (trainer/checkpoint-*, *_result.json, adapter files). Recover: Stop-Process -Id <pid> -Force (PowerShell; use $procId, $id: is a parse error) → device-matmul health check → relaunch with a lower max_length (memory pressure wedges gfx1200).

Environment (measured, don't rediscover)

• GPU: RX 9060 XT gfx1200, 15.92 GiB, Razer Core X TB3, native-Windows ROCm.
• Trainer venv python: C:\Users\nsdha\.unsloth\studio\unsloth_studio\Scripts\python.exe (torch 2.11.0+rocm7.13.0 / hip 7.13.99004 / Unsloth 2026.7.4 / bnb rocm714).
• Training requires a vcvars64 shell (Triton JIT): write a CRLF .bat that calls C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat then runs the venv python; launch from Git Bash as cmd.exe //c '<single-quoted-path>' (double slash beats MSYS path-mangling; never pipe the launch through head). Eager eval (run_eval.py) needs no vcvars.
• llama.cpp ROCm build: ~\.unsloth\llama.cpp\build\bin\Release\ (llama-server.exe, llama-quantize.exe); bundled gguf-py for structural reads. node: ~\.unsloth\node.
• Parallelism 1: one GPU job at a time, batch 1 (+grad accum), workers 0.
• Disk: stage fp16 downloads + merges on D: (HF_HOME=D:\t3-merge\hf, T3_GGUF_DIR=D:\t3-merge\gguf) — ~30 GB peak for a 7B; C: cannot take it.
• Corpus: home clone D:\benny-home (PRIME_SILO_HOME), memo-ray ~\.mem0ray\data (80,555 entities), big LONGVIEW workspace D:\benny-home\benny\workspaces\sessions_v1 (376 JSON cards + data_out prose).