prepare-dataset

Arguments

• $FILE : path to the source file (e.g. data/products.csv, data/clients.json)
• $GOAL : one sentence describing what the model should be able to do after fine-tuning (e.g. "answer questions about our product catalogue in English")

Steps

1. Analyse the source file

Start by reading and analysing $FILE:

• Detect the format (CSV, JSON, JSONL, TSV, Parquet) and the delimiter if CSV
• Identify all available fields/columns and their types (string, number, bool...)
• Count the total number of rows
• Measure the missing value rate per field
• Display 3 sample rows to understand the structure
• Identify rich fields (few missing values, varied content) vs. sparse fields (>50% empty, constant values)

2. Propose a strategy

Based on $GOAL and the analysis, propose:

• Which fields to use and why (exclude fields that are too empty or irrelevant)
• How many Q&A pairs per source row can be generated (~5 to 30 depending on richness)
• Estimated final dataset size (rows x average examples)
• If the dataset is too small (<100 final examples): suggest an augmentation strategy (phrasing variants, synthetic data)
• Recommended filter (e.g. "keep only rows where field X is filled")

Wait for validation before generating the script.

3. Generate scripts/dataset/<n>.py

Generate a Python script that inherits from BaseDatasetPreparer (see scripts/dataset/base.py), with the following structure:

Header and imports

"""
scripts/dataset/<n>.py
──────────────────────────
Short description of what this script does.
Source : $FILE
Goal   : $GOAL

Usage :
    uv run sg-<n> --input data/<n>.csv --stats
    uv run <n>-quality
"""

Lookup tables (if codes or abbreviations are present in the data) Generate Python dicts to translate them into human-readable text (e.g. country codes -> names).

Class <n>Preparer(BaseDatasetPreparer) with two methods:

generate_examples(self, row: dict) -> list[dict]

• Takes one source row as input
• Returns a list of dicts {"instruction": str, "input": str, "output": str}
• Generates between 5 and 30 Q&A pairs per row depending on data richness
• Covers each important fact under 2-3 different phrasings (e.g. "What is X of Y?" / "Give me X for Y." / "What is the X value of Y?")
• Includes at least one "full record" question that aggregates all fields
• Handles missing fields gracefully (if field: before using)
• All questions and answers match the language implied by $GOAL