Skip to content

ADR-0008 — Redis Streams as the broker; no Kafka

  • Status: accepted
  • Date: 2026-07-02
  • Domain: streaming infrastructure · resource sizing

Context

The streaming path needs a broker with real streaming semantics: append-only log, consumer groups, acknowledgement, replay, backlog measurement. The host is a single 2 vCPU / 4 GB VPS already running the full lab stack. Kafka — the reflex choice — wants ~1 GB+ for itself (JVM, plus controller/metadata overhead even in KRaft mode). Redpanda is lighter but still hundreds of MB and brings an operational surface this lab doesn't need. NATS JetStream is tiny but a less recognized name and a different consumption model.

Decision

Redis Streams, single Redis instance, maxmemory capped (~48 MB), streams bounded with XADD MAXLEN. It provides everything the exhibits actually use: consumer groups (XREADGROUP), pending-entry lag (XPENDING/stream length), acknowledgement, and bounded retention. The same instance serves as the buffer for the streamer's Wikimedia EventStreams SSE consumer (CP10, ADR-0015), shared state for run-window deadlines (CP10), and the coupling demo's Hacker News async backlog (CP12) — one dependency, three jobs.

You don't need Kafka to need streaming. The lesson Comparison 1 teaches is about freshness vs cost, and it survives — arguably improves — when the broker costs 40 MB instead of 1 GB.

Consequences

  • Streaming semantics demonstrated at a fraction of the RAM; the whole media stack fits the box.
  • No Kafka on the CV via this project — accepted: the ADR itself demonstrates the more senior skill of not deploying Kafka when it isn't warranted.
  • Redis persistence is weaker than a commit log; acceptable because bronze truth lives in Postgres and the stream is a transport, not the system of record.

Alternatives considered

  • Kafka — rejected: blows the RAM budget; operational weight teaches nothing extra here.
  • Redpanda — rejected: lighter than Kafka but still the heaviest option on the table.
  • NATS JetStream — viable and tiny; rejected on familiarity and because Redis was already earning its keep as shared state.