Skip to content

#616 Part 2: period dashboards move to BigQuery โ€‹

Status: implemented in the same PR as this spec. Issue: #616 (Part 1, the analytics_events mirror drop, shipped in #737). Pattern followed:bqMetrics.service.js (already live for the admin overview + venue rankings).

Why this is a bug fix, not just a cost refactor โ€‹

Every period function in metrics.service.js filters lanterns/waves on createdAt, but lantern docs carry litAt and wave docs carry sentAt (verified in lanternService.js / waveService.js doc shapes). Firestore range filters on a missing field match nothing, so getVenueAnalytics, getMerchantDashboard, getEngagementMetrics (and the already-superseded getPlatformOverview / getVenueRankings) have been returning ZERO period numbers. There is therefore no output-parity constraint: current outputs are wrong, and the migration defines the correct semantics fresh.

Decisions โ€‹

  • D1: delete, don't keep, the superseded Firestore variants.getPlatformOverview and getVenueRankings were replaced by their *Bq twins on the live routes; the Firestore versions (and their now-unused helpers/caches) are removed rather than left dormant.
  • D2: getEngagementMetrics and getVenueAnalytics migrate to BQ; getMerchantDashboard goes hybrid. Venue membership (venueId -> merchantId) exists only in Firestore, so the merchant dashboard keeps its cheap venues lookup, takes lanterns/uniqueUsers per venue from BQ (trusted lens), and keeps waves from the operational waves docs with the sentAt filter fixed: wave EVENTS carry entity_type='wave' with no venue attribution (same constraint buildVenueRankingsSql documents), while wave DOCS carry venueId. The waves scan is bounded (merchant's venues only, period-capped, 5-min cache).
  • D3: trusted lens everywhere counts are spoof-sensitive. Lantern/wave counts filter service_id IS NOT NULL exactly like the live overview and rankings SQL (#596 M-ENTITYID). Consequence, stated openly: merchant/venue lantern numbers read 0 until a server-signed lantern_lit producer ships, which is the same posture the admin platform dashboard already ships with (and the screens showed 0 before this change anyway, per the bug above). Distinct-user ACTIVITY signals (dau/wau/mau) stay unfiltered, matching active_users in the overview SQL.
  • D4: getOfferAnalytics stays on Firestore, documented as such. It reads current-state counters (claimCount/redemptionCount) off a merchant's own offer docs: current state, small bounded scan, not a period aggregation (its period param only labels the response). Migrating it to events would REBUILD those numbers from offer_claimed/offer_redeemed with weaker guarantees than the server-authoritative counters (#694).
  • D5: accept rate = accepted / (accepted + declined) (resolved-set ratio), inheriting the overview's fix for the >100% legacy denominator.
  • D6: retention is unchanged by design. BQ partitions expire at 90 days like the operational purge, so this migration buys cost/load isolation, not longer history: aligned with the prefer-losing-data axiom. Raising BQ retention is a separate, deliberate decision if ever wanted.

Function-by-function โ€‹

FunctionSource afterNotes
getRealtimeActivity / getVenueRealtimeActivityFirestore (unchanged)Current-state snapshots; explicitly out of scope per the issue
getPlatformOverviewDELETEDgetPlatformOverviewBq is the live implementation
getVenueRankingsDELETEDgetVenueRankingsBq is the live implementation
getEngagementMetricsBQdau/wau/mau = distinct user_id over 1/7/30d windows (one query, COUNTIF over window bounds); actions = trusted lantern_lit / wave_sent counts for the period; accept rate per D5
getVenueAnalyticsBQ + bounded waves docstrusted lantern count, distinct users (privacy-guarded), peakHours = EXTRACT(HOUR FROM timestamp) top-5, dailyTrend = DATE(timestamp) series; waves per D2
getMerchantDashboardHybrid per D2venues from Firestore; per-venue lanterns/uniqueUsers from one BQ GROUP BY over entity_id IN UNNEST(@venueIds); waves from docs with sentAt
getOfferAnalyticsFirestore (unchanged)D4; comment updated to say current-state-by-design

Caching keys keep their existing names (values change source); BQ-backed entries move to the bq: prefix convention only where new keys are created.

Built with VitePress