nemo-curator

NeMo Curator - GPU-Accelerated Data Curation

NVIDIA's toolkit for preparing high-quality training data for LLMs.

When to use NeMo Curator

Use NeMo Curator when:

• Preparing LLM training data from web scrapes (Common Crawl)
• Need fast deduplication (16× faster than CPU)
• Curating multi-modal datasets (text, images, video, audio)
• Filtering low-quality or toxic content
• Scaling data processing across GPU cluster

Performance:

• 16× faster fuzzy deduplication (8TB RedPajama v2)
• 40% lower TCO vs CPU alternatives
• Near-linear scaling across GPU nodes

Use alternatives instead:

• datatrove: CPU-based, open-source data processing
• dolma: Allen AI's data toolkit
• Ray Data: General ML data processing (no curation focus)

Quick start

Installation

# NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized):
#   text-cuda12 / text-cpu (and image/video/audio/math variants), or `all`.

# Text curation (CUDA 12)
uv pip install "nemo-curator[text-cuda12]"

# All modalities
uv pip install "nemo-curator[all]"

# CPU-only text (slower)
uv pip install "nemo-curator[text-cpu]"

Basic text curation pipeline

Major version rewrite (1.x): NeMo Curator was rewritten around a Ray-based pipeline/stage architecture. The old DocumentDataset + nemo_curator.modules.* / ScoreFilter / Modify call-the-object-on-a-dataset API from 0.x is gone. In 1.x you compose ProcessingStages into a Pipeline and run it with an executor. The exact stage/import surface differs per modality — treat the examples in this skill below as conceptual (0.x-style) and follow the current quickstart and text guide for the exact 1.x APIs rather than copying imports verbatim.

Shape of a 1.x pipeline (from the upstream quickstart):

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.base import ProcessingStage
from nemo_curator.stages.resources import Resources
from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.core.client import RayClient

# 1. Define/compose stages (load -> filter -> dedupe -> classify -> write).
#    Each stage declares its own Resources (CPU cores, GPU memory, replicas).
pipeline = Pipeline(name="curation", stages=[...])

# 2. Run it with an executor (Ray-backed).
client = RayClient()
client.start()
pipeline.run(XennaExecutor())
client.stop()