dataops-and-rag-architectures

DataOps & RAG Architectures

Core Competencies & Directives

Implement CI/CD for data pipelines utilizing lakeFS concepts. Write Python code that utilizes zero-clone branching for isolated dev/test data environments and automated hooks for pre-commit data validation.

When designing Retrieval-Augmented Generation (RAG) pipelines, favor deterministic ingestion and structured storage as the foundational layer, using vector databases strictly as a fuzzy recall fallback rather than a pure vector search approach.

When to Use

• designing lakeFS-style branch/merge workflows for datasets
• writing pre-commit data validation hooks
• building RAG ingestion with structured primary storage
• preventing context poisoning in agent retrieval layers
• isolating dev/test data environments without full copies

Workflow

1. Branch isolation — model environments as lakeFS branches (main, dev/{feature}, test/{run}).
2. Pre-commit hooks — schema, volume, and contract checks before branch commit (see hooks/data-validation-guard.sh).
3. Deterministic ingestion — normalize to structured tables/documents with stable IDs and provenance.
4. Vector fallback — index embeddings only for fuzzy recall; primary answers come from structured lookup.
5. Validate — references/rag-deterministic-ingestion-checklist.md.

RAG Layer Model

Structured store (source of truth)
    ↓ deterministic lookup by ID / key / filter
Vector index (fuzzy recall fallback)
    ↓ rerank + provenance attach
Agent context (signed, TTL-bound)

Coding Constraints

• When writing Python, include comprehensive docstrings, static typing, and defensive error handling.
• Never output legacy, manual deployment scripts; everything must be "as code".
• Ensure multi-tenant data isolation with RBAC/ABAC on branches and buckets.

Anti-Patterns

• pure vector search with no structured canonical store
• shared dev/test branches without isolation
• ingestion without provenance metadata
• skipping pre-commit validation "for small datasets"