factory-data-pipelines

Factory data pipelines

Each section leads with Principle (one sentence, stack-agnostic), then Why (constraint → option → tradeoff), then Recipe (the Papa Parse / FastAPI / Cloud Run shape we use), and Failure mode when there's one to name. Sections that are pure style with no deeper truth are marked Recipe only.

CSV ingestion — a script, not a framework

Principle. A one-shot CSV importer is a standalone script in scripts/; promote to a framework only when a second consumer appears.

Why. Premature job-framework adoption for a single CSV importer is paying the abstraction cost without the abstraction benefit. A standalone script with Papa Parse and a Drizzle transaction is grep-able, debuggable, runnable locally. Wrapping it in BullMQ / Inngest / a Cloud Run job buys nothing until there's a second importer that shares the wrapper.

Recipe.

scripts/
└── data_processing/
    ├── import-state-data.ts
    ├── simulations/
    │   └── import-simulation.ts
    └── vins/
        └── import-vins.ts

import Papa from 'papaparse';
import { db } from '@/db';
import { foo } from '@/db/schema';

const { data } = Papa.parse<FooRow>(csvText, { header: true, skipEmptyLines: true });

await db.transaction(async (tx) => {
  for (const row of data) {
    await tx.insert(foo).values({ /* ... */ }).onConflictDoUpdate({ /* ... */ });
  }
});

Time-series / event storage — JSONB envelope

Principle. What drives queries gets a real column. What doesn't goes in a JSONB envelope. The schema inside JSONB can evolve without a migration.

Why. Time-series data and event streams have an outer schema (the row) and an inner schema (the payload). The outer schema needs to be queryable — filter by fleetId, sort by time, join across rows. The inner schema is consumed by application code that already speaks its types, doesn't need a SQL index, and may evolve faster than migrations can keep up. JSONB envelope splits these clean: columns for outer, JSONB for inner. The trap is querying inside JSONB at app speed — once you're doing that, the field has earned a column.