grpo-rl-training

Kheish Compatibility

This skill is repo-local and stays inactive until explicitly activated.

When the original instructions refer to legacy tool names, use these Kheish mappings:

• terminal => bash
• web_extract => web_fetch, plus web_search when discovery is needed
• search_files => grep_search and glob_search
• browser_* tools require a browser-capable surfaced tool or MCP; if none is available, use the closest available surface and say so explicitly

When the instructions mention local helper files, resolve them from ${KHEISH_SKILL_DIR}.

GRPO/RL Training with TRL

Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions.

When to Use This Skill

Use GRPO training when you need to:

• Enforce specific output formats (e.g., XML tags, JSON, structured reasoning)
• Teach verifiable tasks with objective correctness metrics (math, coding, fact-checking)
• Improve reasoning capabilities by rewarding chain-of-thought patterns
• Align models to domain-specific behaviors without labeled preference data
• Optimize for multiple objectives simultaneously (format + correctness + style)

Do NOT use GRPO for:

• Simple supervised fine-tuning tasks (use SFT instead)
• Tasks without clear reward signals
• When you already have high-quality preference pairs (use DPO/PPO instead)

---

Core Concepts

1. GRPO Algorithm Fundamentals

Key Mechanism:

• Generates multiple completions for each prompt (group size: 4-16)
• Compares completions within each group using reward functions
• Updates policy to favor higher-rewarded responses relative to the group

Critical Difference from PPO:

• No separate reward model needed
• More sample-efficient (learns from within-group comparisons)
• Simpler to implement and debug