Skip to content

ADR-0015 — Wikimedia EventStreams + Hacker News replace Twitch as the media pipeline's data sources

  • Status: accepted
  • Date: 2026-07-04
  • Domain: ingestion architecture · data sources

Context

The media pipeline's three comparative showcases (batch vs streaming, sync vs async, Postgres vs DuckDB) were built around Twitch Helix as a single "media engagement" source. Twitch's developer console registration is now confirmed permanently blocked for this project's maintainer (Morocco) — not a transient outage, so waiting it out is not a real option.

Beyond the registration block, a design flaw surfaced on re-evaluation: Twitch Helix is poll-only (no push API for viewer counts — this was ADR-0007's whole premise). The "streaming" side of Comparison 1 would only ever have been simulated via high-frequency polling, which undercuts the very tradeoff the showcase exists to make observable.

The replacement must satisfy, without compromise: - No auth friction — this is a solo project on a budget VPS; no app review, OAuth dance, or fragile key issuance. - Real streaming semantics — the batch-vs-streaming showcase should contrast a genuine push protocol against interval-based batch, not two flavors of polling. - A natural fan-out shape — the sync-vs-async showcase needs many small independent HTTP requests so the async speedup (and, per the CP12 design, the cascade-vs-absorb fault-injection story) is honest and dramatic. - Analytical volume — enough event volume for the Postgres-vs-DuckDB comparison to be meaningful within weeks, not months. - Stability — public, documented, long-lived endpoints; no ToS ambiguity for a public portfolio repo. - Fit within the capacity budget (docs/plan/capacity-budget.md) on the Hetzner VPS.

Decision

Dual source, each mapped to the showcase it serves best, rather than forcing one source to do everything:

Showcase Source Why it fits
Batch vs streaming (CP10) Wikimedia EventStreams (recentchange, SSE) A true push protocol vs interval-windowed capture of the same firehose — a real contrast, not simulated.
Sync vs async (CP12) Hacker News Firebase API /topstories + per-item /item/{id} forces one-request-per-item hydration — a natural fan-out of hundreds of small GETs; sequential await vs asyncio.gather + semaphore yields an honest, measurable benchmark, and gives the fault-injection hook a real call site to sit behind.
Postgres vs DuckDB (CP13) Both EventStreams volume reaches analytical scale within days; HN engagement snapshots (score/comment deltas over repeated observations) add a mutable-fact dimension.

Zero API keys anywhere in the ingestion layer as a result — a genuine security simplification, not just a swap.

Consequences

  • The streaming demo now uses a real SSE firehose; the honesty story improves — "Twitch is poll-only" becomes a documented rejected alternative (ADR-0007, now superseded by this record) rather than a limitation the exhibit had to work around.
  • No secrets anywhere in the ingestion layer — simpler CI, simpler .env, smaller public-repo attack surface.
  • Two sources instead of one means two ingestion modules to build and monitor, mitigated by shared plumbing (Postgres sink, Redis buffering, one bronze contract, common idempotency pattern).
  • The original "creator economy" / media-viewership narrative is lost; HN partially preserves an engagement-metrics angle (scores, comment velocity). docs/plan/media-pipeline-context.md needs an ingestion-section revision — the comparison-axis reasoning itself (freshness-vs-cost, coupling-vs-resilience, row-vs-column store) is source-independent and stays as written.
  • EventStreams is an unbounded firehose: requires backpressure handling and a retention/filtering policy to respect the VPS capacity budget. Tripwire: sustained ingest lag or Postgres bloat triggers filtering to a wiki subset (via WIKI_STREAM_URL), not vertical scaling.
  • SSE connections drop periodically (documented Wikimedia behavior) — the streamer must implement Last-Event-ID resume to avoid silent gaps.
  • The sync-vs-async showcase now makes real outbound calls to a third-party API from a publicly-triggerable endpoint — a new risk class versus the original fully-synthetic design; concurrency and rate must be capped regardless of fault state (see CP12's security review).

Alternatives considered

  • GitHub Events API + GH Archive — a legitimate runner-up: GH Archive's hourly dumps vs the Events API is a textbook batch/near-real-time contrast, and archive volume flatters DuckDB. But the Events API is still polling (fails the real-streaming requirement), and archive volume risks the capacity budget without aggressive filtering. Kept as a candidate future extension, not the backbone.
  • YouTube Data API v3 — closest to the original media-engagement framing, and registration is low-friction (a Google Cloud API key, not gated like Twitch's dev console). Rejected as the backbone: its 10,000-units/day quota reshapes the architecture around quota management instead of concurrency and streaming, weakening exactly the properties this decision needs. Acceptable as a future low-volume enrichment source.
  • Reddit API — rejected: auth-hostile since the 2023 API changes, OAuth required for anything useful, real ToS risk for a public portfolio.
  • Mastodon public timelines — rejected: a real streaming API with no auth on many instances, but low volume on most of them and policy variability instance-to-instance.
  • Wait for Twitch registration to recover — rejected: unbounded timeline on a block that is confirmed permanent, and even on success the poll-only limitation would remain.
  • Wikimedia EventStreams docs: https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams
  • Hacker News API: https://github.com/HackerNews/API
  • Supersedes the Twitch-based source selection in docs/plan/media-pipeline-context.md and ADR-0007.