dataset-curator

Dataset Curator

Overview

This skill covers the full lifecycle of dataset creation and curation for machine learning and LLM tasks. It addresses dataset schema design, data collection strategies, quality filtering, deduplication, class imbalance mitigation, stratified train/val/test splits, annotation guideline writing, and dataset card documentation. Good datasets are the foundation of reliable models — this skill helps teams avoid the most common data quality pitfalls that lead to poor generalization, evaluation leakage, and biased models.

When to Use

• Designing a new dataset schema for a classification, extraction, or generation task
• Cleaning and deduplicating a raw dataset before model training
• Planning annotation guidelines for human labelers or LLM-assisted labeling
• Addressing class imbalance in a training set (oversampling, undersampling, weighting)
• Creating stratified train/val/test splits without leakage between splits
• Writing a dataset card (model card for data) for reproducibility and documentation
• Auditing an existing dataset for quality, coverage, and potential biases
• Combining multiple data sources into a single unified dataset

When NOT to Use

• Training or fine-tuning a model (use model training skills)
• Running SQL or analytical queries on a production database (use data analysis skills)
• Building real-time data pipelines or streaming ETL (use data engineering skills)
• Designing evaluation suites for deployed LLMs (use eval-designer skill)
• Web scraping or data collection from APIs (use data collection skills)

Quick Reference

Task	Approach
Define dataset schema	List fields, types, required vs optional, allowed values, and examples
Remove duplicates	Hash-based exact dedup + MinHash/LSH for near-duplicate detection
Fix class imbalance	Oversample minority (SMOTE) or undersample majority; adjust loss weights
Create train/val/test splits	Stratified split by label; ensure no overlap of entities across splits
Document the dataset	Write a dataset card with provenance, schema, statistics, and limitations
Validate annotation quality	Compute inter-annotator agreement (Cohen's kappa or Krippendorff's alpha)
Handle missing values	Decide per-field: impute, drop row, or add "unknown" category
Detect label noise	Use confident learning (cleanlab) or cross-validation outlier detection