nanopath: Instructions file for Codex

Project goals:

• Nanopath should be easy to share with collaborators so they can quickly try new training objectives, preprocessing choices, data curation ideas, and hyperparameters on a small model.
• The normal loop is: iterate fast on a single H100, validate promising changes with downstream probes, then reserve larger/full-node training for the best candidates.
• Keep the codebase hackable and nanochat-like: flat organization, few files, few lines, and minimal abstractions.

Before changing code:

• If not already activated, source nanopath's .venv.
• For broad or ambiguous tasks, read deeply enough into the current repo to understand the training/probing/data path before recommending changes. Look at every relevant source, config, script, and doc file rather than optimizing one file in isolation.
• Make a concrete multi-step plan for nontrivial work, then keep going through implementation, validation, and any needed doc/comment updates.
• Default to immediately implementing sensible recommendations and validating them rather than simply suggesting recommendations.

Coding guidelines:

• Use flat organization: as few folders, files, and lines of code as possible; functional elegance is the goal. If you make revisions where there are over a dozen new lines of code I am going to be highly skeptical you really tried your best to adhere to this guideline. A great revision should LOWER the total lines of code, not increase it. Don't play smart by opening subprocesses or other hacks to get around this limitation.
• Commenting is the exception to the line-count preference: add concise comments explaining "how and why" for functions and important (i.e., not plotting/logging) code blocks.
• Do not add defensive try/except blocks or fallbacks. If something is wrong, it should fail loudly. Don't bother with ValueError raises or other code guards.
• Prefer hard-coded constants over extra environment variables, modular options, or fallback paths, unless the value is meant to be frequently tuned.
• Prefer native PyTorch over Accelerate, Lightning, etc. nanopath is single-GPU only — do not introduce DDP, FSDP, or any multi-GPU code path.
• Do not use argparse. Meaningful tunables should live in YAML config files, e.g. cfg.train.lr; if YAML does not define a variable used by a training script, it is fine for that to error. Only put variables in YAML when they are actually meant to be tuned often; otherwise hard-code them.
• Avoid tiny helper functions/classes that are only a handful of lines. Put the code directly where it is used.
• Follow nanochat as the model for a clean minimalist codebase, especially train.py and model.py.
• Do not create new files unless explicitly asked or truly necessary; prefer improving existing files. If you do create a new file, add a few commented out lines of code to the top of it to explain its purpose.
• If code changes make comments, docs, configs, or scripts inaccurate, update those too.

Experiment and benchmark discipline:

• Validate opinions experimentally whenever feasible. Run code, tests, probes, or short jobs that directly support the conclusion.
• Use downstream probing as the main comparison signal because objectives like JEPA, MAE, DINO, and iBOT may not have comparable validation losses.
• Treat one run as discovery. A maintainer reruns promising candidates three times with different random training seeds; the median run becomes the validated leader if it beats the incumbent by at least 0.004. robust-norm-s9876 is the approved exception.
• Use wandb for logging, plotting, and utilization monitoring throughout pretraining. Log all metrics needed to validate training behavior (i.e., gradient norm).
• Full runs launched with ./submit/train_1gpu.sbatch ... prompt for Labless run name, notes, and GitHub no-scope device login before scheduling, then auto-submit after a successful eligible run.
• For direct python train.py runs or frozen baseline evaluations worth sharing, run ./labless/submit_to_labless.py output_dir=... run_name=... notes=...; labless records the verified GitHub login. Full submissions require summary.json, metrics.jsonl, summary.max_train_samples == 1000000, summary.tile_presentations <= 1000000, and summary.max_train_flops == 1e18. Keep smoke checks and failed runs local.
• Labless notes should name the starting recipe, main changes, and why those changes might affect probes; agents should derive them from the source/config diff.
• Do not submit runs whose saved labless_source snapshot changes probe.py or benchmarking/; labless marks locked-path changes invalid.