dataset-builder

Computer Vision Dataset Builder

Purpose

Create reproducible CV dataset artifacts from owned or licensed media. Make provenance, class definitions, annotation formats, split strategy, validation, augmentation, versioning, and privacy decisions explicit before training.

Use When

• Extracting frames from video or images from an approved source.
• Defining or converting YOLO, COCO, or another documented annotation format.
• Creating train/validation/test splits, class maps, dataset manifests, or quality reports.

Do Not Use When

• Media ownership, consent, license, or intended use is unknown.
• The task is model training/inference without dataset changes.
• The request asks to scrape private media, bypass access controls, or infer sensitive labels.

Inputs

• Source inventory, ownership/consent/license, retention, and allowed uses.
• Dataset task, class ontology, inclusion/exclusion rules, and annotation format.
• Sampling policy, duplicate policy, split groups, ratios, seed, and version ID.
• Image constraints, labeling tool/export source, augmentation policy, and destination outside the Hipson repository.

Default Stack

• Python, OpenCV, NumPy, and standard JSON/CSV/YAML parsers in a project-owned environment.
• Local filesystem and content hashes for the baseline; Roboflow, Supervision, and Hugging Face remain optional integrations.
• YOLO or COCO only when the downstream model contract requires it.
• Group-aware deterministic splits and augmentation after splitting.

Workflow

1. Inventory sources and record owner, license, consent, collection method, allowed use, retention, and sensitive-content handling. Stop if rights are unresolved.
2. Freeze a versioned class map and annotation policy with positive, negative, ignored, occluded, truncated, and ambiguous examples.
3. Extract frames deterministically by timestamp, interval, or scene rule. Use collision-safe identifiers derived from stable source IDs, never private paths.
4. Validate decoding, dimensions, channels, corrupt files, exact/near duplicates, empty labels, unknown classes, normalized coordinate bounds, and polygon shape.
5. Split by source group, subject, site, session, or video before augmentation to prevent adjacent frames or related subjects from leaking across partitions.
6. Apply augmentation only to training data and record the transform policy, random seed, tool/library version, and whether labels were transformed.
7. Convert formats through one deterministic adapter and round-trip a sample. Preserve the canonical source annotations separately.
8. Write a manifest with hashes, counts, class distribution, split statistics, rejected items, schema version, and unresolved quality risks.
9. End with vision-verifier; do not claim dataset suitability from file counts alone.