datasets

Datasets: Loading and Processing Data

Overview

Hugging Face Datasets provides a library for easily loading and processing datasets from the Hub, local files, or in-memory data. Apply this skill for loading datasets, preprocessing, tokenization, caching, and efficient data handling for machine learning.

When to Use This Skill

This skill should be used when:

• Loading datasets from Hugging Face Hub
• Loading datasets from local files (CSV, JSON, Parquet, etc.)
• Tokenizing text data for transformers
• Processing large datasets efficiently
• Creating train/validation/test splits
• Caching processed datasets
• Working with memory-mapped datasets
• Streaming large datasets

Quick Start

Basic Import and Setup

from datasets import load_dataset, Dataset, DatasetDict

Loading Datasets

# Load dataset from Hub
dataset = load_dataset("glue", "mrpc", split="train")
print(dataset)
# Dataset(features: {'idx': Value(dtype='int32', id=None), 'label': ClassLabel(num_classes=2, names=['not_equivalent', 'equivalent'], id=None), 'sentence1': Value(dtype='string', id=None), 'sentence2': Value(dtype='string', id=None)}, num_rows: 4084)

# Load specific split
train_dataset = load_dataset("glue", "mrpc", split="train")
val_dataset = load_dataset("glue", "mrpc", split="validation")

# Load entire dataset with all splits
dataset = load_dataset("glue", "mrpc")
# DatasetDict({
#     train: Dataset(features: {...}, num_rows: 4084)
#     validation: Dataset(features: {...}, num_rows: 4084)
#     test: Dataset(features: {...}, num_rows: 4084)
# })

Loading from Local Files

# From CSV
dataset = load_dataset("csv", data_files="train.csv", split="train")

# Multiple files
dataset = load_dataset(
    "csv",
    data_files=["train1.csv", "train2.csv"],
    split="train"
)

# From JSON
dataset = load_dataset("json", data_files="data.json", field="data")

# From Parquet
dataset = load_dataset("parquet", data_files="train.parquet")

# From text
dataset = load_dataset("text", data_files="data.txt")