AReaL CLAUDE.md

CLAUDE.md - AReaL

WHAT: Project Overview

AReaL is a distributed RL training framework for LLM alignment via reinforcement learning.

Tech Stack: Python 3.12+ | PyTorch | FSDP2/Megatron | SGLang/vLLM

Core Directories:

• areal/ - Core package
  • api/ - Config dataclasses, workflow/engine contracts
  • engine/ - FSDP2, Megatron, SGLang/vLLM adapters
    • fsdp_utils/ - FSDP2-specific utilities (checkpoint, grad, optimizer, parallel)
    • megatron_utils/ - Megatron/FP8 utilities (checkpoint, pipeline, quantization)
    • core/ - Engine-shared utilities (distributed, lock, model, offload)
  • infra/ - Infrastructure (launcher, scheduler, RPC)
    • utils/ - Infrastructure utilities (launcher, proc, http, concurrent, slurm, ray)
  • workflow/ - RolloutWorkflow implementations
  • reward/ - Reward functions
  • dataset/ - Dataset loaders
  • utils/ - Cross-cutting utilities (logging, data, checkpoints, network, RL functional)
• examples/ - Training scripts and configs
• docs/ - Jupyter Book source

WHY: Purpose

• Enable efficient RL training for LLM alignment at scale
• Async rollout + distributed training for high throughput
• Modular design: workflows, engines, rewards, and datasets are independently extensible

HOW: Core Commands

# Check environment
python --version              # Requires 3.12+
uv --version                  # Install: https://docs.astral.sh/uv/

# Sync dependencies
uv sync --extra cuda          # CUDA + SGLang inference (default)
# For vLLM: cp pyproject.vllm.toml pyproject.toml && cp uv.vllm.lock uv.lock && uv sync --extra cuda
uv sync --group dev           # Include dev/test packages
uv run python3 areal/tools/validate_installation.py  # Validate installation

# Pre-commit hooks
pre-commit install --install-hooks  # Set up hooks (run once)
pre-commit run --all-files    # Format and lint

# Run tests
# First check GPU availability (many tests require GPU)
python -c "import torch; print('GPU available:', torch.cuda.is_available())"
uv run pytest tests/test_<topic>.py

# Generate CLI docs
uv run python docs/generate_cli_docs.py

# Build docs (canonical, release-aligned)
./docs/build_all.sh
# Do NOT use `jupyter-book build docs/en|docs/zh` directly for final preview/release,
# because it skips AReaL-specific static setup and output packaging.