data-lake-architecture

Data Lake Architecture

Purpose

Decide where analytical data lives and how it is laid out so that it stays cheap, queryable by several engines, governed, and does not turn into a "data swamp". Modern designs keep data in open formats (Parquet + an open table format) on object storage, with a catalog and access control, and attach compute engines as needed.

Agent Protocol

Trigger

"data lake", "lake vs warehouse", "lakehouse", "s3/gcs/adls layout", "bronze/silver/gold", "medallion", "partitioning", "small files", "iceberg vs delta vs hudi", "storage-compute separation".

Input Context

• Sources (OLTP/CDC, events, files, SaaS), volumes and growth, latency needs (batch, hourly, near real time).
• Consumers: BI, data science/ML, reverse ETL, external sharing.
• Cloud/vendor constraints, existing warehouse, governance/compliance (PII, residency, deletion requests).

Output Artifact

Architecture decision + layer definitions + layout conventions + engine/catalog choice. No file unless requested.

Response Format

Pattern: {warehouse|lake|lakehouse} because {reason}
Storage: {bucket layout} format={Parquet + Iceberg/Delta/Hudi}
Layers: bronze={...} silver={...} gold={...}
Catalog/governance: {catalog, access control, PII handling}
Engines: {Spark/Trino/Flink/warehouse}

Completion Criteria

• Decision recorded with alternatives (ADR).
• Every table has an owner, layer, partition spec and retention.
• PII identified and access-controlled; deletion path exists (GDPR/CCPA).
• File sizing/compaction plan to avoid small files.

Warehouse vs Lake vs Lakehouse

	Warehouse	Data lake (files only)	Lakehouse (open table format)
Storage	Managed, proprietary	Object storage, raw files	Object storage, Parquet + table metadata
Schema	On write	On read	On write with schema evolution
Transactions / updates	Yes	No (rewrite files)	Yes: ACID commits, MERGE/UPDATE/DELETE, time travel
Engines	The warehouse	Many, inconsistent	Many engines on the same tables
Best for	BI-centric teams, SQL-only, low ops	Raw archive, ML on unstructured data	Mixed BI + ML + streaming at scale, avoiding lock-in
Many organizations combine them: lakehouse for raw/curated layers, warehouse (which now often reads Iceberg/Delta directly) for serving.