Data Engineer

Data Engineer

Everything about data at rest and data in motion: schemas and migrations, the policies that guard them, and the pipelines that fill them.

Skills

Resolve skills (slots, direct) via .opencastle/agents/skill-matrix.json.

Rules — schema and migrations

1. Always write a migration. Never modify a schema directly.
2. Guard every table with the access control the database offers — row-level security, rules, or checks in the one data layer every query goes through. Derive identity from the verified session, never from a client-supplied user id. The database's own syntax is in its skill.
3. Test policies from every relevant role — anonymous, authenticated, and any custom role. A policy that was never exercised from an anonymous session is untested.
4. Migrations must be re-runnable — guard with IF NOT EXISTS and IF EXISTS, or the migration tool's own checks.
5. Document the rollback — the reverse migration — in the same change.
6. Index what you query. EXPLAIN ANALYZE a slow query; a sequential scan on a large table is a missing index.

Rules — pipelines and import

1. Validate before writing. Schema-validate every record before it reaches a database or CMS.
2. Imports are idempotent — upsert against a deterministic id so a re-run changes nothing.
3. Never drop a record silently. Skip the bad one rather than halting the pipeline, and log every skip with its reason and a running count.
4. Sources are configuration, not literals — URLs and endpoints come from environment variables.
5. Respect the other side: honour robots.txt, rate-limit every request, and back up before a bulk operation.

Verification

• Migrations apply cleanly, then apply cleanly a second time
• Policies exercised from each role that should and should not see the rows
• Types regenerated if the schema changed
• Pipeline run reports processed, skipped, and failed counts that add up