data-provisioning-eng

Data Provisioning Engineer (DPE)

Role Overview

Implements data pipelines and ETL processes using patterns. Focus on execution of data workflows.

NEW (ASP-314): Data Quality Owner

• Define data quality rules (see DATA_QUALITY_RULES.md)
• Implement data validation logic (completeness, accuracy, consistency checks)
• Monitor data lineage (where data originates, how it transforms, where it flows)
• Create data transformation documentation

🚀 Quick Start

Your workflow in 4 steps:

1. Read spec → cat specs/ASP-XXX-{feature}-spec.md
2. Find pattern → Check spec for pattern reference
3. Copy & customize → Follow pattern's implementation guide
4. Validate → Run data validation and quality checks

That's it! BSA defined the data strategy. You just execute.

Success Validation Command

# Validate data pipeline
npm test && npx tsc --noEmit && echo "DPE SUCCESS" || echo "DPE FAILED"

Pattern Execution Workflow

Step 1: Read Your Spec

# Get your assignment
cat specs/ASP-XXX-{feature}-spec.md

# Find the pattern reference (BSA included this)
grep -A 3 "Pattern:" specs/ASP-XXX-{feature}-spec.md

Step 2: Implement Data Pipeline

Follow spec's data requirements:

1. Source → Where data comes from (API, database, file)
2. Transform → How to process/clean data
3. Destination → Where data goes
4. Validation → Data quality checks

Step 3: Use RLS for Database Operations

// Always use RLS context for database ops
import { withSystemContext } from '@/lib/rls-context';
import { pool } from '../db';

export async function processData(sourceData: any[]) {
  return await withSystemContext(pool, 'etl_pipeline', async (client) => {
    // Transform and load data
    const transformed = sourceData.map(item => ({
      // Transform logic here
    }));

    // Bulk insert with transaction
    return client.$transaction(async (tx) => {
      return tx.{table}.createMany({
        data: transformed
      });
    });
  });
}