kaggle-data-format-first

Kaggle Data Format Verification Before Research

Problem

Competition names and file sizes can be misleading. Investing in RAG research, technical planning, or model architecture design before verifying the actual data format leads to significant wasted effort when assumptions don't match reality.

Real example:

• Competition: "vesuvius-challenge-surface-detection"
• Expected (from RAG): 3D TIFF stacks, TopoScore, 128³ patches
• Actual data: 2D grayscale images (320×320), binary masks
• Waste: Hours of RAG research on wrong problem

Context / Trigger Conditions

Use this skill when:

• Starting ANY new Kaggle competition
• Competition name is ambiguous about data dimensionality (2D vs 3D)
• Data size suggests one format but could be another
• Planning to do RAG research or extensive technical planning
• File extensions are generic (.tif, .png, .npy could be anything)

Red flags:

• Competition name mentions "3D", "volume", "surface" but you haven't verified
• Large download size (>5GB) but unsure what format it actually is
• Multiple data directories with unclear purpose (train_images vs train vs train_data)

Solution

Phase 1: Quick Format Check (Before ANY Research)

Step 1: Download only a sample first

# If possible, download just one file to verify format
# Or download full data but check structure immediately

kaggle competitions download -c {competition-slug}
unzip {competition-file}.zip

Step 2: Verify data structure in <5 minutes

import os
from PIL import Image
import numpy as np

# Quick check script
data_dir = "path/to/unzipped/data"

# What files exist?
print("Directories:", [d for d in os.listdir(data_dir) if os.path.isdir(os.path.join(data_dir, d))])
print("Files:", [f for f in os.listdir(data_dir) if os.path.isfile(os.path.join(data_dir, f))][:10])

# Check dimensions of first sample
samples = []
for root, dirs, files in os.walk(data_dir):
    for f in files:
        if f.endswith(('.tif', '.png', '.jpg', '.npy')):
            path = os.path.join(root, f)
            if f.endswith('.npy'):
                data = np.load(path)
            else:
                data = np.array(Image.open(path))

            print(f"Sample: {f}")
            print(f"  Shape: {data.shape}")
            print(f"  Dtype: {data.dtype}")
            print(f"  Range: [{data.min()}, {data.max()}]")

            samples.append({
                'path': path,
                'shape': data.shape,
                'dtype': str(data.dtype)
            })

            if len(samples) >= 3:
                break
    if len(samples) >= 3:
        break

# Determine data type
if all(len(s['shape']) == 2 for s in samples):
    print("✓ Data Type: 2D Images")
elif all(len(s['shape']) == 3 for s in samples):
    print("✓ Data Type: 3D Volumes")
else:
    print("⚠ Mixed or irregular dimensions")