alterlab-pufferlib

PufferLib - High-Performance Reinforcement Learning

Overview

PufferLib is a high-performance reinforcement learning library designed for fast parallel environment simulation and training. It achieves training at millions of steps per second through optimized vectorization, native multi-agent support, and efficient PPO implementation (PuffeRL). The library provides the Ocean suite of 20+ environments and seamless integration with Gymnasium, PettingZoo, and specialized RL frameworks.

When to Use This Skill

Use this skill when:

• Training RL agents with PPO on any environment (single or multi-agent)
• Creating custom environments using the PufferEnv API
• Optimizing performance for parallel environment simulation (vectorization)
• Integrating existing environments from Gymnasium, PettingZoo, Atari, Procgen, etc.
• Developing policies with CNN, LSTM, or custom architectures
• Scaling RL to millions of steps per second for faster experimentation
• Multi-agent RL with native multi-agent environment support

Core Capabilities

1. High-Performance Training (PuffeRL)

PuffeRL is PufferLib's optimized PPO trainer (CleanRL-derived, with optional LSTM via models.LSTMWrapper) built for high-throughput training.

Recommended path — CLI / high-level helper. Drive training from a config (an .ini in pufferlib/config/) rather than hand-wiring the trainer:

# CLI: env name resolves to a registered config + Ocean env
puffer train puffer_breakout --train.device cuda --train.learning-rate 0.015

import pufferlib.pufferl as pufferl
# train(env_name, args=None, vecenv=None, policy=None, logger=None)
pufferl.train('puffer_breakout')

Manual loop. PuffeRL(config, vecenv, policy, logger=None) — note the first arg is a config dict (not flat kwargs), the env arg is vecenv, and the loop is driven by global_step. The three loop methods are real: evaluate(), train(), mean_and_log().

import pufferlib.vector
from pufferlib.pufferl import PuffeRL, load_config

# Native PufferEnv -> default backend=PufferEnv. For wrapped (Gymnasium/
# PettingZoo) envs you MUST pass backend=pufferlib.vector.Multiprocessing.
vecenv = pufferlib.vector.make(MyPufferEnv, num_envs=256)

# load_config returns a nested args dict (sections: 'train', 'vec', 'env', ...)
# with defaults from pufferlib/config/*.ini. PuffeRL takes the 'train' section.
args = load_config('puffer_breakout')
config = {**args['train'], 'env': 'puffer_breakout'}
config['device'] = 'cuda'

trainer = PuffeRL(config, vecenv, my_policy)
while trainer.global_step < config['total_timesteps']:
    trainer.evaluate()      # Collect rollouts
    trainer.train()         # Train on batch
    trainer.mean_and_log()  # Aggregate + log