data-quality-audit

Skill: data-quality-audit

Purpose

Audit training data and tokenizer pipelines for quality issues that silently degrade LLM training. Data problems are the #1 underdiagnosed cause of poor model performance — most tutorials skip this entirely.

Trigger

Use when:

• Preparing a dataset for LLM pretraining or fine-tuning
• Model outputs seem degenerate (repetition, wrong language, truncated responses)
• Tokenizer was recently changed or vocabulary was updated
• Merging datasets from multiple sources

Background (Why This Matters)

From LLM-from-scratch practitioners:

"I spent two weeks debugging a model that kept outputting garbled text. The issue was 0.3% of my training data had HTML entities that the tokenizer split into hundreds of tokens, dominating the loss landscape."

Key lessons:

1. Token length distribution matters — outlier-length documents dominate gradient updates disproportionately
2. Deduplication is not optional — even 1% duplicate data causes memorization artifacts
3. Tokenizer fertility (chars/token) should be consistent — sudden drops indicate encoding bugs
4. BOS/EOS tokens must be consistent — missing end tokens cause the model to never learn to stop
5. Language distribution — unlabeled multilingual data causes unexpected behavior in monolingual models

Usage

/data-quality-audit [dataset_path_or_glob] [--tokenizer model_name_or_path] [--sample 10000]

Checks Performed

1. Token Length Distribution

P5:   [tokens]
P50:  [tokens]  
P95:  [tokens]
P99:  [tokens]
MAX:  [tokens]  ← flag if >> context_window

WARNING: Documents at P99+ length will be truncated, losing their tail content.
         Consider splitting or filtering documents > 0.8 * context_window.

2. Deduplication Check

• Exact match on first 64 tokens (cheap, catches near-duplicates)
• MinHash LSH for near-duplicate detection (sampled)
• Reports estimated duplicate % in dataset

3. Tokenizer Fertility Check

# chars_per_token should be stable across document types
# English prose: ~4.0 chars/token
# Code: ~3.0-3.5 chars/token
# Sudden drop to <2.0 suggests tokenizer is splitting on noise
fertility = total_chars / total_tokens