apache-kafka-event-streaming-and-schema-registry

Apache Kafka Event Streaming & Schema Registry Guide

1. Overview & Core Philosophy

Apache Kafka is an append-only distributed commit log designed for high-throughput, fault-tolerant event streaming. Unlike traditional message queues (e.g., RabbitMQ) that delete messages upon consumer ACK, Kafka retains messages for a configurable retention window, allowing multiple heterogeneous consumer groups to read and re-read event streams independently.

Key Architectural Axioms

• Log Ordering Guarantee: Total order is guaranteed only within a single partition, not across partitions in a topic.
• Partitioning as Unit of Parallelism: Maximum concurrency for a consumer group is bounded by the number of partitions in a topic.
• Decoupled State via Schema Registry: Producers and consumers must share data contracts via explicit schemas (Avro, Protobuf, or JSON Schema) registered in a Schema Registry rather than untyped payloads.
• Zero-Data-Loss Default: Production setups must enforce strict durability constraints (acks=all, min.insync.replicas=2, replication.factor=3).

---

2. Topic Architecture & Partitioning Strategy

2.1 Partition Key Selection & Distribution

Selecting an optimal partition key is critical for balance and ordering.

• Natural Entity Keys: Key by domain entity ID (e.g., order_id, customer_id) when state updates for that specific entity must arrive in order.
• Custom Partitioner: Implement custom hashing (or murmur2 default) to prevent skew when entity keys follow power-law distributions (hotspot keys).
• Null Keys (Round-Robin / Sticky): Use only for high-throughput append logs where ordering across events is irrelevant.

2.2 Topic Configuration Checklist

# Production Durability & Retention Standard
replication.factor=3
min.insync.replicas=2
cleanup.policy=delete # or compact for state tables
retention.ms=604800000 # 7 days
segment.bytes=1073741824 # 1GB
unclean.leader.election.enable=false