kafka-rails

Kafka in Rails

Kafka is the right tool when you have multiple consumers of the same event stream, need replay, or need to retain events for days. Wrong tool for "I want to send a job" — use Solid Queue / Sidekiq. This skill covers production patterns for Rails apps using karafka, the maintained Ruby Kafka client.

The opinion

Use karafka 2.x for both producing and consuming. Use the Confluent Schema Registry with Avro or Protobuf — never raw JSON in production. Pick partition keys by entity (e.g., account_id) so per-entity events are ordered. Consumers are at-least-once by default — make them idempotent. For transactional publish, use the outbox pattern (write to DB + outbox in one transaction, separate process flushes to Kafka).

Counter-positions:

• racecar (Heroku) — simpler, but less actively maintained; lacks karafka's web UI, swarm mode, and DLQ support.
• ruby-kafka — the underlying low-level client; rarely used directly anymore.
• rdkafka-ruby — karafka's underlying high-perf binding to librdkafka. Use directly only if you need very low-latency.

Setup

# Gemfile
gem "karafka", "~> 2.4"
gem "karafka-web"  # admin UI
gem "avro_turf"    # for schema registry; or "google-protobuf"

# karafka.rb (at the project root)
class KarafkaApp < Karafka::App
  setup do |config|
    config.kafka = {
      "bootstrap.servers": ENV.fetch("KAFKA_BROKERS"),
      "client.id": "myapp",
      "compression.type": "zstd"
    }
    config.client_id = "myapp"
    config.consumer_persistence = !Rails.env.development?
  end

  Karafka::Web.enable!

  routes.draw do
    topic "order_events" do
      consumer OrderEventsConsumer
      max_messages 100
      max_wait_time 1_000
      dead_letter_queue topic: "order_events_dlq", max_retries: 5
    end
  end
end

Pattern 1: Producing events (with outbox)

NEVER produce inline. If the broker is down, your business transaction can't commit (or commits with a missing event). Use the outbox pattern — canonical schema defined in event-driven-architecture: