Skip to content

ADR-0006 — Scheduled micro-batch as the media pipeline's default path, streaming built as the priced alternative

  • Status: accepted
  • Date: 2026-06-30
  • Domain: ingestion architecture · resource sizing

Context

The lab's media project ingests engagement data (Wikimedia edit events, Hacker News engagement — see ADR-0015) into a small warehouse and serves trend metrics over an API and dashboard. The analytical questions behind that (rolling windows, daily aggregates, top-mover detection) are trend-shaped: a freshness target of minutes-to-hours, not seconds, is enough to answer them. The decision is the ingestion paradigm for that production-shaped use case: pull data on a schedule (micro-batch), or consume it as a continuous event stream — constrained by the same single-host, no-orchestrator hardware that shapes the rest of the lab (ADR-0001), where a full streaming backbone competes with the warehouse and monitoring stack for the same RAM.

Separately, the project's whole portfolio purpose is to make architecture tradeoffs observable (see docs/plan/media-pipeline-context.md), which means the rejected alternative — streaming — does not stay purely theoretical here: CP10 builds it too, as a live, on-demand comparison, so a viewer can see the freshness-vs-cost gap as real numbers rather than take the tradeoff on faith. This ADR is about which path is the recommended default for this workload, not about which paths get built.

Decision

Scheduled micro-batch is the default, production-shaped ingestion path. Streaming would solve a latency problem this workload's consumers do not have, at a RAM cost the host cannot comfortably spare alongside Postgres and monitoring. A periodic capture window lands raw events in the bronze layer; dbt incremental models build the layers above it. Backfills are idempotent — re-running a window reproduces the same result — which makes recovery and reprocessing trivial.

The streaming path is still built, but as Comparison 1's priced alternative, not as a second production path: a resident-but-idle service that a viewer triggers into a bounded run window (ADR-0009), publishing events through Redis Streams (ADR-0008) as they arrive from a genuine SSE push source (ADR-0015). Its job is to make this ADR's tradeoff observable, not to replace batch as the recommended choice — and because the rejected alternative is now a real push stream rather than a simulated one, the tradeoff it demonstrates is more honest than the pipeline's original Twitch-based design allowed.

Consequences

  • The recommended path fits the RAM budget with no streaming backbone competing for memory at steady state; it is simple, reliable, and easy to reason about, with idempotent backfills making reprocessing safe. dbt incremental models are the natural fit for windowed metric building.
  • It is not real-time — the pipeline is only as fresh as the polling interval, and polling can miss spikes that occur and resolve within an interval. The capture-window length, not source appetite, sets the achievable cadence.
  • The bronze landing layer is defined as a contract (a raw-event schema) independent of how events arrive (docs/contracts/media-bronze-contract.md). The streaming comparison writes into the same bronze layer, so this choice is confined to the ingestion edge and does not propagate into the model layer above it — adding, or later dropping, a streaming source is an additive change, not a redesign. Revisit if a consumer appears that genuinely needs sub-minute freshness in production, or a second source arrives that is natively a stream, making it cheaper to consume as events than to poll.

Alternatives considered

  • Event streaming as the production default — rejected: solves a latency requirement this workload's consumers do not have, at a RAM cost (a full streaming backbone) this host cannot spare at steady state.
  • Hybrid (stream the hot path, batch the rest) as the production default — rejected: adds operational complexity — two ingestion paths to maintain in production — for a freshness need that does not exist here; kept instead as the CP10 on-demand comparison rather than a standing path.