tao-finetune-cosmos-reason

Cosmos3 TAO training

Keep one shared model-facing frontend. Backend image fields and contract paths live under backend_contracts in references/skill_info.yaml; image literals are stamped from versions.yaml, while referenced backend YAMLs define native runtime schemas. Never translate between them.

Mandatory runtime intake

Before planning training, collect all of the following. Do not infer a path from history, another user, a prior job, an image, or a developer checkout.

• base_model_path_or_uri. For a Hugging Face model ID or URL, accept an optional friendly base_model_revision such as a branch or tag. If omitted, resolve main; do not ask the user for a commit SHA. Resolve the selected ref read-only through the Hub API to its immutable commit and seal the model ID, requested ref, and resolved SHA in the plan. A complete local snapshot needs no revision and is sealed by its file fingerprints.
• For Cosmos3-Nano, an explicit input-checkpoint model_type: qwen3_vl or cosmos3_omni. If the user did not supply it, ask once before planning; never infer the choice from config.json, a model ID, a path name, or a previous run. Explain the two choices in plain language: qwen3_vl uses a compatible Hugging Face checkpoint directly, while cosmos3_omni requires an immutable conversion to exact Qwen3-VL safetensors before training. Record the answer as base_model_format. Cosmos3-Edge is inferred as cosmos3_edge from the resolved model ID and does not present this Nano-only choice.
• Do not expose Omni preparation implementation fields during normal intake. For Nano, use Qwen/Qwen3-VL-8B-Instruct, resolve both Hub models to immutable commits, and run the selected backend's packaged converter. Framework uses cosmos_framework.scripts.convert_model_to_vlm_safetensors; Cosmos-RL uses cosmos_rl.model_preparation.vlm_safetensors with its pinned isolated converter runtime. Validate an explicitly supplied prepared_checkpoint_path or donor, but never ask for one by default.
• Accept hf_model://nvidia/Cosmos3-Nano directly. If a gated/private model cannot be resolved, ask the user only to set HF_TOKEN in the session environment; never ask them to discover a SHA or provide the token value in chat.
• explicit video sampling mode: either uniform nframes or fps. FPS mode may also set min_frames and max_frames; both modes may set clip-time, resize, and pixel-budget fields supported by the selected backend.
• training/validation annotation paths and media roots for conversation-style or task-aware video supervision, plus optional task selection.
• explicit backend for a comparison; cosmos-framework or cosmos-rl.
• training_mode; dense or peft. PEFT also requires rank, alpha, dropout, target modules, bias, RS-LoRA, modules-to-save, and adapter precision.
• user-owned results_dir, checkpoint_dir, cache_dir, and, for SLURM, sqsh_cache_dir, ssh_key_path, mounts, and scheduler settings.
• Runtime order: compute-readable sqsh_path, explicit image, then the selected backend image in references/skill_info.yaml. On SLURM reuse or convert it once under sqsh_cache_dir. Never compare an SQSH filename with an image tag or request source provenance/SHA.
• Repository paths, commits/trees, branch, base image, build context, and timestamp are advanced inputs required only for explicit source-build. See references/cosmos-backend-operations.md; never infer a build from runtime selection.