investigate-dataset

Investigate Dataset

This workflow helps you explore and understand datasets used in evaluations. It covers HuggingFace datasets, CSV files, and JSON/JSONL files.

Key Concepts

For detailed information on Inspect's dataset types (datasets.Dataset vs inspect_ai.dataset.Dataset), the hf_dataset() pipeline, caching behaviour, and test utilities, see references/inspect-dataset-patterns.md.

Common Patterns in Evals

Evals typically define:

• DATASET_PATH: HuggingFace repo path (e.g., "qiaojin/PubMedQA")
• DATASET_REVISION: Optional git revision/tag for reproducibility
• record_to_sample(): Function converting raw records to Sample objects

Prerequisites

• Access to the evaluation code to find dataset configuration
• Python environment with datasets, pandas, and inspect_ai installed

Steps

1. Identify the Dataset Source

Look for these patterns in the evaluation code:

# HuggingFace dataset
DATASET_PATH = "org/dataset-name"
DATASET_REVISION = "v1.0"  # optional
hf_dataset(path=DATASET_PATH, name="subset", split="train", ...)

# CSV dataset
csv_dataset("path/to/file.csv", ...)
load_csv_dataset("https://example.com/file.csv", eval_name="myeval", ...)

# JSON/JSONL dataset
json_dataset("path/to/file.json", ...)
load_json_dataset("https://example.com/file.jsonl", eval_name="myeval", ...)

2. Load the Raw Dataset

For investigation, load the raw data directly (not through Inspect's sample_fields transformation). Use standard datasets.load_dataset() for HuggingFace, pd.read_csv() for CSV, or pd.read_json() for JSON/JSONL. For gated datasets, ensure HF_TOKEN is set or run huggingface-cli login.

3. Explore Structure and Quality

Use standard pandas/datasets methods to explore:

• Schema: ds.features (HF) or df.dtypes (pandas)
• Shape: len(ds), ds.column_names (HF) or df.info(), df.columns (pandas)
• Sample data: ds[:3] (HF) or df.head() (pandas)
• Missing values: Check for None, empty strings, empty lists
• Duplicates: Check ID uniqueness if an ID field exists
• Value distributions: value_counts() for categorical columns, length stats for text fields