scientific-data-preprocessing

Scientific Data Preprocessing Skill

⚠️ CRITICAL: USER'S HARD-WON EXPERIENCE - MANDATORY CONSULTATION ⚠️

This skill encapsulates painful lessons learned from real preprocessing disasters (88.9% error rate documented). ALWAYS use this skill for planning, reflection, and validation when ANY data preprocessing is involved.

Why this skill is mandatory:

• Based on actual project failures (V1.0, V2.0 case studies)
• Prevents data leakage that causes production disasters
• Catches semantic errors AI agents commonly make
• Saves weeks of debugging and model retraining

When to invoke (DO NOT SKIP):

• ✅ Before starting ANY data preprocessing task
• ✅ During preprocessing for reflection and validation
• ✅ After preprocessing for comprehensive audit
• ✅ When reviewing AI-generated preprocessing code

---

Core Mission

Prevent catastrophic preprocessing errors in grouped time-series data by applying multi-level feature analysis and respecting data structure boundaries.

When to Use This Skill

MANDATORY consultation - trigger immediately when:

Data Preprocessing Tasks (ALWAYS)

• Any data cleaning, transformation, or preparation work
• Loading and preparing data for modeling
• Creating training/test splits
• Handling missing values (imputation, deletion)
• Feature scaling/normalization/standardization
• Encoding categorical variables
• Feature engineering or construction
• Feature selection or dimensionality reduction

Data Structure Types (ALWAYS)

• Preprocesssing time-series data with natural groupings (matches, sessions, patients, experiments)
• Sports analytics (tennis, basketball, etc.)
• Medical/clinical data with patient groupings
• Panel data or longitudinal studies
• Any grouped/hierarchical data structure

Quality Assurance (ALWAYS)

• Auditing existing preprocessing for data leakage or semantic errors
• Reviewing AI-generated preprocessing code for common pitfalls
• Validating preprocessing before model training
• Debugging unexpected model performance