data-engineering

Data Engineering

Overview

Data engineering is the foundation of every data-driven decision. Bad data pipelines silently corrupt analytics, break ML models, and lead to business decisions based on false premises. This skill covers designing pipelines that are correct, observable, and resilient — from ingestion to serving.

The data engineering contract: Every pipeline must guarantee that what lands in the destination is what the source intended, or it must fail loudly. Silent data corruption is the worst failure mode.

When to Use

• Building ETL, ELT, or streaming data pipelines
• Designing data warehouse schemas (star, snowflake, data vault)
• Migrating data between systems or formats
• Setting up data quality monitoring and anomaly detection
• Creating CDC (change data capture) pipelines
• Building feature stores for machine learning
• Handling schema evolution without breaking consumers

NOT for:

• Simple one-off data exports (use a script)
• Real-time systems with sub-second latency requirements (use stream processing)
• Data science / analysis work (this skill is about moving and transforming data, not interpreting it)

The Data Pipeline Process

Step 1: Define the Data Contract

Before writing any pipeline code, define what correctness means:

DATA CONTRACT:
├── Source: [system, table, API, file format]
├── Destination: [system, table, format]
├── Schema: [field names, types, nullability, defaults]
├── Volume: [records/day, peak throughput, growth rate]
├── Latency: [batch hourly / batch daily / streaming / near-real-time]
├── Quality rules: [uniqueness, referential integrity, range checks]
├── Retention: [how long to keep, compliance requirements]
└── SLA: [acceptable downtime, max lag, error rate threshold]

Schema definition example:

# data_contract.yaml
source:
  system: production_postgres
  table: orders
  
destination:
  system: snowflake
  schema: analytics
  table: fact_orders
  
schema:
  order_id: { type: BIGINT, nullable: false, unique: true }
  customer_id: { type: BIGINT, nullable: false }
  order_date: { type: TIMESTAMP, nullable: false }
  amount: { type: DECIMAL(10,2), nullable: false, min: 0 }
  status: { type: VARCHAR(20), nullable: false, enum: [pending, paid, shipped, cancelled] }
  
quality_rules:
  - column: order_id
    check: not_null
  - column: amount
    check: range
    min: 0
    max: 100000
  - table: fact_orders
    check: referential_integrity
    references: dim_customers.customer_id