Analytics pipeline hardening (consolidated plan)

Date: 2026-07-27 Issues consolidated: #613, #614, #615, #616, #617, #618, #691 (refs #596, #315, #626) Status: SHIPPED. All four phases merged and deployed 2026-07-28; live tables verified. Riders in #742 (done), successor project #740 in flight. Skills consulted: planning, map produced by a full pipeline trace (this doc's section 2)

1. How to read this doc

Section 2 is the frozen baseline (what the pipeline actually does today, traced 2026-07-27). Section 3 is the phase plan; section 5 is the live dashboard, updated as phases land. One phase = one PR = one session (rule 12). The issue bodies carry the detailed designs; this doc adds what they could not: cross-issue sequencing, the tracing discoveries that change their assumptions, and the two decisions that need the operator.

2. Baseline: the pipeline as traced (frozen)

One hot path serves everything: apps/web/src/lib/flash.js (in-memory buffer, flush at 10 events / 30s / tab-hide) to services/api/analytics/src/routes/tracking.js to tracking.service.js (batch fans out into N individual calls) to packages/forge/index.js to firestore.js (analytics_events mirror) + bigquery.js (one streaming insert per event).

Discoveries from the trace that CHANGE the issues' assumptions:

• The Firestore mirror has zero working readers. Its only consumer, the admin taxonomy discover scan (admin.js:369-375), queries a timestamp field the mirror docs do not have (they carry createdAt), so it returns nothing and has for some time. #616 Part 1 is therefore even safer than the issue argues: dropping the mirror breaks nobody, and un-breaks event discovery when the query is repointed at BigQuery.
• The mirror write is the path's only throw-on-failure. forge/index.js:111-117 fails the whole track call on a Firestore hiccup while a BigQuery failure is a warn. Dropping the mirror removes a whole failure mode, not just cost.
• analytics.events IS day-partitioned already (setup-bigquery.sh:87-89, 90-day partition expiry) contrary to #617's suspicion; what is missing is clustering and require_partition_filter.
• event_counts_daily clusters on a dead column: created with --clustering_fields=event_type, but the schema and MERGE use event_name (renamed later). The clustering is inert.
• A 429-conflation bug: tracking.js:179 returns 429 for generic TRACKING_FAILED, and 429 is the only status Flash re-buffers on. Real server-side write failures are thus disguised as rate limits and re-sent forever instead of being counted as drops. Belongs to #618 Part 1.
• Page-close flush is dead code: the sendBeacon branch can never carry auth headers and every path falls through to a cancellable plain fetch (flash.js:684-697). #614's keepalive quick-win is accurately scoped.
• Trust split confirmed (#691): lantern_lit/wave_sent are server-signed by lanterns-api; wave_accepted/wave_declined are client-emitted after client-side Firestore writes, spoofable, and deliberately counted unfiltered in bqMetrics.service.js:91-97.

3. Phases (each is one PR)

Phase 1: stop paying for dead weight (#616 Part 1 + #617 Part 1)

Drop the Firestore mirror from Forge; repoint the (currently broken) taxonomy discover at BigQuery; make trackBatch a real batch (one BigQuery insert([rows]) per request instead of N). No schema changes, immediate cost and latency win, removes the throw-on-Firestore failure mode. Update analytics-infrastructure.md and the EventCreator copy. Deletes forge/firestore.js from the hot path and the dead getCollection() export.

Also in Phase 1, an audit prompted by the operator's two-store question (2026-07-27): the nightly FIRESTORE rollup layer (analytics_daily_venue / analytics_daily_platform / analytics_daily_cost / weekly / monthly, written by the daily-aggregation scheduled job) appears to have no data reader; the only admin touchpoint found is the aggregation-status HEALTH check in BigQueryWorkspace, which reports which dates ran, not the numbers. If the audit confirms no reader, retire the job and collections in Phase 1 (point the health view at BQ bq-export-status coverage instead); if a reader surfaces, migrate it to event_counts_daily first. Operational-collection reads for the REALTIME dashboard (lanterns/waves "right now") are explicitly kept: that is the correct half of the realtime-from-Firestore principle, not dead weight.

Phase 2: client-stamped context + table hygiene (#613 + #617 Part 2)

Thread event_time (coarse, whole seconds) and session_id from Flash through the Zod schema, Forge, and the BigQuery schema; keep server receive-time as the separate trusted field. Then the table work in the same PR (it needs the schema change anyway): add clustering (event_name, user_id) to analytics.events, set require_partition_filter, and fix event_counts_daily's inert event_type clustering to event_name. Verify the LIVE tables first (the setup scripts may have drifted from what exists).

Phase 3: stop losing data silently (#614 + #618 Part 1)

The keepalive fetch replaces the dead sendBeacon branch (small, could even ride Phase 2 if trivial); the IndexedDB durable queue re-hydrates on init (depends on Phase 2's event_time). Server side: drop/error counters with structured logs, fix the 429-conflation so real failures return 5xx and get counted (client dead-letters them into the durable queue rather than hot-looping), and alert on drop-rate.

Phase 4: trust alignment (#691, scoped down)

Add count_trusted per-trust columns to the event_counts_daily MERGE so the raw-volume and trusted lenses both exist without ambiguity (#691 item 2b). Revisit forge's per-user budget vs venue_viewed volume (#691 item 3) with real Phase 1-3 telemetry. #691 item 1 (server-signed accept/decline) is deliberately NOT here; see decision D2.

4. Decisions needed from the operator

D1 (#615, pre-login tracking): recommend DEFER, then option C. Signup is invite-gated, so today's pre-login funnel is a handful of known invitees; the measurement value is near zero while the privacy surface (any durable pre-login identity) is real. Recommendation: defer the decision until public signup is on the roadmap, and when it is, take option C (ephemeral per-visit id, no durable identity) per the prefer-losing-data axiom; option A (anonymous auth) retroactively links pre-signup behavior to the eventual real account, which is exactly the kind of quiet linkability the axioms exist to prevent. Close #615 as deferred-with-direction once confirmed.

D2 (#691 item 1, server-signed wave accept): recommend treating as its own future project, not part of this cluster. Making wave_accepted trustworthy means moving acceptWave (a client-side Firestore write that also creates the connection doc, with rules built around recipient-created connections and E2EE key exchange) into the lanterns API. That is a wave-domain migration with a privacy-architecture review, not an analytics change; bolting it onto this cluster would blow the phase budget. It also naturally pairs with any future decline-related server work from the #731/D14 model. Recommend: file it as its own issue when Phase 4 lands, wired to rule 9's migrate-on-touch principle.

5. Live dashboard

Phase	Issues	Status
1	#616 P1, #617 P1, rollup retirement	MERGED (#737) + hotfix (#738); deployed
2	#613, #617 P2, #614 quick-win	MERGED (#739); table settings applied + verified live 2026-07-28
3	#614, #618 P1	MERGED (#741) incl. all five ultrareview fixes
4	#691 (2b), #618 P2 deferred	MERGED (#741): count_trusted live; D2 successor #740 in flight; #618 P2 (durable limiter) deferred pending telemetry
D1	#615	DECIDED by operator 2026-07-27: direction C (ephemeral per-visit id, no durable pre-login identity); implementation deferred until public signup approaches. #615 closed with the decision
D2	#691 item 1	APPROVED by operator 2026-07-27: split out as future wave-domain project; shipped as #740 (PR #743), merged + deployed 2026-07-30
5 (close-out)	#618 P1 residue + P2 decision	Done 2026-07-30: drop-rate alert policy added to setup-analytics-alerts.sh, client 429/disabled drop paths counted, /health writeFailures documented in openapi, #270 closed as already-shipped

#618 Part 2 (durable rate limiter) DECIDED 2026-07-30: accept-per-instance-with-alerting. Rationale: the track routes are double-gated (App Check + Firebase auth) and forge applies its own per-user budget; a Firestore transaction per track request would tax the hottest write path for a soft limit; Memorystore is real infrastructure for a pilot. The looseness is bounded and observable instead: --max-instances=20 is pinned on the analytics-api deploy (worst case = 20 x maxEventsPerMinute per user, cost-trivial at BigQuery row prices), and every 429 now emits a structured analytics_rate_limited log. Revisit a shared store only if that log shows sustained multi-instance abuse. Decision recorded in services/api/analytics/src/middleware/rateLimiter.js.