retail-eda-framework

Retail/Fashion EDA Framework

Comprehensive EDA approach for retail/fashion/tabular data, validated through Walmart, H&M, and 7+ fashion-lifecycle-pricing competitions.

Problem

Most ML projects skip EDA or do it superficially (just df.describe() + a histogram). This leads to:

• Missing data issues discovered too late
• Train/test distribution shift not caught
• High-cardinality features used as raw categoricals (overfit)
• Text features not extracted from product descriptions
• Time-based leakage in train/test split
• Hierarchical structure ignored (e.g., 5-level category hierarchy in H&M)

Real case (validated 2026-06-02 on H&M articles.csv):

• 105,542 rows × 25 cols
• Only 416 missing values (0.39% in detail_desc)
• 5-level hierarchical structure: index_group → section → department → product_type → product_group
• 45,875 unique prod_name (high cardinality)
• 43,404 unique detail_desc (text, 142 chars avg)

Without proper EDA, none of these are caught before feature engineering.

The 5-Stage EDA Pipeline

Stage 1: Data Quality Audit (FIRST)

Tools: ydata-profiling (13.5k★), missingno (4.2k★)

# Quick data quality report
import ydata_profiling
profile = ydata_profiling.ProfileReport(df, title="Data Quality Report")
profile.to_file("eda/data_quality.html")

# Missing data visualization
import missingno as msno
msno.matrix(df)        # Bar chart of missing per column
msno.heatmap(df)       # Correlation of missingness between columns
msno.dendrogram(df)    # Hierarchical clustering of missingness

Look for:

• Missing value patterns (random vs systematic)
• High-cardinality categoricals (will overfit tree models)
• Skewed numerical features (need log transform)
• Constant/quasi-constant features (drop immediately)
• Duplicate rows
• Outliers (use IQR or z-score, not just visual)

Stage 2: Statistical Profiling (Train vs Test)

Tools: sweetviz (3.1k★)

# Compare train vs test
import sweetviz as sv
report = sv.compare([train_df, "Train"], [test_df, "Test"], target_feat="target")
report.show_html("eda/train_vs_test.html")