redis-streams-rails

Redis Streams

Redis Streams is the "I already have Redis, give me Kafka-ish semantics without a new operational dependency" option. Good up to medium throughput, single-region, and short retention. Outgrows the pattern at high throughput / long retention — that's Kafka territory.

The opinion

Use Redis Streams when you already run Redis (you almost certainly do) and you need pub/sub-with-history + consumer groups, but don't yet justify Kafka's operational cost. Cap streams with MAXLEN. Use consumer groups with manual acknowledgement. Periodically claim entries idle for > N minutes (PEL handling). Plan to migrate to Kafka if throughput exceeds ~10k events/sec sustained or you need cross-region replication.

Why Redis Streams over plain pub/sub

Feature	Redis pub/sub	Redis Streams
Persistence	None (drop on publish if no subscriber)	Disk + AOF
Replay	None	Yes (read from any offset)
Consumer groups	No	Yes
Acknowledgement	No	Yes (XACK)
Pending tracking	No	Yes (XPENDING)
Memory growth	None	Bounded with MAXLEN

If you need anything beyond fire-and-forget broadcasts: Streams, not pub/sub.

Setup

# Gemfile
gem "redis", "~> 5.0"
gem "connection_pool"  # for thread-safe sharing

# config/initializers/redis.rb
REDIS_STREAMS = ConnectionPool.new(size: 10, timeout: 5) do
  Redis.new(url: ENV.fetch("REDIS_STREAMS_URL"))
end

Pattern 1: Producing (XADD)

class EventPublisher
  STREAM = "orders.events"
  MAX_LEN = 1_000_000  # approximate cap

  def self.publish(event_type:, payload:)
    REDIS_STREAMS.with do |redis|
      redis.xadd(
        STREAM,
        {
          event_type: event_type,
          payload: payload.to_json,
          published_at: Time.current.iso8601
        },
        maxlen: MAX_LEN,
        approximate: true  # ~ trim — Redis trims when convenient, much faster
      )
    end
  end
end

EventPublisher.publish(
  event_type: "order.placed",
  payload: { order_id: 42, account_id: 7, total_cents: 12_500 }
)