data-modeling

Data Modeling

When users present a data modeling request, start from data understanding and complete the full pipeline—from exploration to output. Run fully automated end-to-end; never pause to ask the user for intermediate decisions.

Core Principles

• Strictly adhere to task requirements regarding features, preprocessing methods (if specified), and output format.
• All preprocessing parameters must be fitted on the training set and consistently transformed on the test set.
• Prioritize efficient methods and models from lightweight libraries like scikit-learn to quickly validate feasibility.
• Always consider data scale: for large sample sizes or high-dimensional features, implement OOM safeguards (e.g., sampling, streaming processing, avoiding full One-Hot encoding, limiting tree depth, etc.).
• Ensure outputs are well-structured and logically reproducible; avoid data leakage or dimension mismatches.
• 🚫 No plotting allowed: You cannot view charts. All feature analysis must be performed through computed statistics only.

Data Exploration

Perform rapid preliminary analysis on the provided data:

• Check shape: number of samples and features.
• Identify feature types:
  • Numerical (continuous/discrete)
  • Categorical (nominal/ordinal)
  • Temporal (datetime)
  • Special fields: text, IDs, high-cardinality features, etc.

Implementation Workflow

Modeling Strategy: Start simple, iterate progressively.

✅ Phase 1: Quick Baseline (Mandatory)

Use simple, efficient methods and models to establish an end-to-end pipeline and generate an initial prediction:

• Essential Preprocessing:
  • Handle missing values: impute with reasonable defaults (e.g., mean, median, mode) based on feature type.
  • Drop irrelevant columns: e.g., IDs, UUIDs, serial numbers, or other unique identifiers.
  • Encoding:
    • Numerical features: standardize if needed.
    • Categorical features:
      • High-cardinality categories (unique values ≥ 10): Avoid One-Hot encoding to prevent dimension explosion. Use compact representations such as Label Encoding, Frequency Encoding, Target Encoding, or Hash Encoding. Choose the encoding method based on data characteristics and model compatibility.
      • Low-cardinality categories (unique values < 10): One-Hot encoding may be acceptable, but evaluate total feature count—if many categorical columns exist, prefer low-dimensional alternatives to avoid feature inflation.