data-eng-streaming-patterns

Streaming Data Patterns

Real-time data processing patterns for event-driven systems and streaming pipelines.

Core Principles

1. Events are immutable facts -- Never mutate events in-flight; produce corrective events instead.
2. Design for failure and replay -- Every component must handle reprocessing gracefully through idempotent writes and offset management.
3. Exactly-once is a system property -- It emerges from idempotent producers, transactional writes, and deterministic processing combined.
4. Backpressure over data loss -- When consumers fall behind, slow down producers or buffer intelligently rather than dropping messages.
5. Schema evolution is mandatory -- Plan for backward and forward compatible schema changes from day one using a schema registry.

Apache Kafka Fundamentals

Kafka organizes data into topics split into partitions. Producers write by key for ordering; consumers read in groups for load balancing.

from confluent_kafka import Producer
import json

producer = Producer({
    "bootstrap.servers": "broker1:9092,broker2:9092",
    "enable.idempotence": True,
    "acks": "all",
    "compression.type": "lz4",
})

def produce_clickstream_event(user_id: str, event: dict):
    """Partition by user_id to preserve per-user ordering."""
    producer.produce(
        topic="clickstream-events",
        key=user_id.encode("utf-8"),
        value=json.dumps(event).encode("utf-8"),
        callback=lambda err, msg: logger.error(f"Failed: {err}") if err else None,
    )
    producer.poll(0)

See kafka-patterns.md for:

• Consumer group configuration and rebalancing strategies
• Partitioning strategies for high-cardinality keys
• Exactly-once producer transactions
• Topic configuration and retention policies

Change Data Capture (CDC) Patterns

CDC captures row-level changes from database transaction logs, avoiding polling overhead.

# Debezium PostgreSQL connector
name: "orders-connector"
config:
  connector.class: "io.debezium.connector.postgresql.PostgresConnector"
  database.hostname: "postgres-primary"
  database.dbname: "inventory"
  table.include.list: "public.orders,public.customers"
  plugin.name: "pgoutput"
  slot.name: "debezium_slot"
  tombstones.on.delete: true
  snapshot.mode: "initial"
  transforms: "route"
  transforms.route.type: "org.apache.kafka.connect.transforms.RegexRouter"
  transforms.route.regex: "([^.]+)\\.([^.]+)\\.([^.]+)"
  transforms.route.replacement: "cdc.$3"