huggingface-llm-trainer

TRL Training on Hugging Face Jobs

Overview

Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required — models train on cloud GPUs and results are automatically saved to the Hugging Face Hub.

TRL provides multiple training methods:

• SFT (Supervised Fine-Tuning) — standard instruction tuning
• DPO (Direct Preference Optimization) — alignment from preference data
• GRPO (Group Relative Policy Optimization) — online RL training
• Reward Modeling — train reward models for RLHF

See references/training_methods.md for method overviews and selection guidance.

When to Use Unsloth

Use Unsloth (references/unsloth.md) instead of standard TRL when GPU memory is limited (~60% less VRAM), speed matters (~2x faster), training large models (>13B), or training Vision-Language Models (Unsloth has FastVisionModel support). See scripts/unsloth_sft_example.py for a production-ready training script.

Key Directives

1. Submit jobs via hf jobs uv run (CLI) or the hf_jobs() MCP tool if the Hugging Face MCP server is configured — pass the training script inline, don't save to a local file unless the user explicitly requests it. If the user asks to "train a model" or "fine-tune", create the training script AND submit the job immediately.
2. Always include Trackio for real-time monitoring — use scripts/ templates.
3. Provide job details after submission: job ID, monitoring URL, estimated time; note the user can request status checks later.
4. Use example scripts as templates: scripts/train_sft_example.py, scripts/train_dpo_example.py, etc.

Local Script Execution

Repository scripts use PEP 723 inline dependencies. Run them with uv run:

uv run scripts/estimate_cost.py --help
uv run scripts/dataset_inspector.py --help

Prerequisites Checklist

Account & Authentication:

• Hugging Face account with Pro/Team/Enterprise plan (Jobs require a paid plan); authenticated login.
• HF_TOKEN for Hub push is CRITICAL — the training environment is ephemeral, so results are lost unless pushed to the Hub. Token must have write permissions. Pass secrets={"HF_TOKEN": "$HF_TOKEN"} in the job config.