#616 Part 2: period dashboards move to BigQuery โ
Status: implemented in the same PR as this spec. Issue: #616 (Part 1, the analytics_events mirror drop, shipped in #737). Pattern followed:bqMetrics.service.js (already live for the admin overview + venue rankings).
Why this is a bug fix, not just a cost refactor โ
Every period function in metrics.service.js filters lanterns/waves on createdAt, but lantern docs carry litAt and wave docs carry sentAt (verified in lanternService.js / waveService.js doc shapes). Firestore range filters on a missing field match nothing, so getVenueAnalytics, getMerchantDashboard, getEngagementMetrics (and the already-superseded getPlatformOverview / getVenueRankings) have been returning ZERO period numbers. There is therefore no output-parity constraint: current outputs are wrong, and the migration defines the correct semantics fresh.
Decisions โ
- D1: delete, don't keep, the superseded Firestore variants.
getPlatformOverviewandgetVenueRankingswere replaced by their*Bqtwins on the live routes; the Firestore versions (and their now-unused helpers/caches) are removed rather than left dormant. - D2:
getEngagementMetricsandgetVenueAnalyticsmigrate to BQ;getMerchantDashboardgoes hybrid. Venue membership (venueId -> merchantId) exists only in Firestore, so the merchant dashboard keeps its cheapvenueslookup, takes lanterns/uniqueUsers per venue from BQ (trusted lens), and keeps waves from the operationalwavesdocs with thesentAtfilter fixed: wave EVENTS carryentity_type='wave'with no venue attribution (same constraintbuildVenueRankingsSqldocuments), while wave DOCS carryvenueId. The waves scan is bounded (merchant's venues only, period-capped, 5-min cache). - D3: trusted lens everywhere counts are spoof-sensitive. Lantern/wave counts filter
service_id IS NOT NULLexactly like the live overview and rankings SQL (#596 M-ENTITYID). Consequence, stated openly: merchant/venue lantern numbers read 0 until a server-signedlantern_litproducer ships, which is the same posture the admin platform dashboard already ships with (and the screens showed 0 before this change anyway, per the bug above). Distinct-user ACTIVITY signals (dau/wau/mau) stay unfiltered, matchingactive_usersin the overview SQL. - D4:
getOfferAnalyticsstays on Firestore, documented as such. It reads current-state counters (claimCount/redemptionCount) off a merchant's own offer docs: current state, small bounded scan, not a period aggregation (itsperiodparam only labels the response). Migrating it to events would REBUILD those numbers fromoffer_claimed/offer_redeemedwith weaker guarantees than the server-authoritative counters (#694). - D5: accept rate = accepted / (accepted + declined) (resolved-set ratio), inheriting the overview's fix for the >100% legacy denominator.
- D6: retention is unchanged by design. BQ partitions expire at 90 days like the operational purge, so this migration buys cost/load isolation, not longer history: aligned with the prefer-losing-data axiom. Raising BQ retention is a separate, deliberate decision if ever wanted.
Function-by-function โ
| Function | Source after | Notes |
|---|---|---|
| getRealtimeActivity / getVenueRealtimeActivity | Firestore (unchanged) | Current-state snapshots; explicitly out of scope per the issue |
| getPlatformOverview | DELETED | getPlatformOverviewBq is the live implementation |
| getVenueRankings | DELETED | getVenueRankingsBq is the live implementation |
| getEngagementMetrics | BQ | dau/wau/mau = distinct user_id over 1/7/30d windows (one query, COUNTIF over window bounds); actions = trusted lantern_lit / wave_sent counts for the period; accept rate per D5 |
| getVenueAnalytics | BQ + bounded waves docs | trusted lantern count, distinct users (privacy-guarded), peakHours = EXTRACT(HOUR FROM timestamp) top-5, dailyTrend = DATE(timestamp) series; waves per D2 |
| getMerchantDashboard | Hybrid per D2 | venues from Firestore; per-venue lanterns/uniqueUsers from one BQ GROUP BY over entity_id IN UNNEST(@venueIds); waves from docs with sentAt |
| getOfferAnalytics | Firestore (unchanged) | D4; comment updated to say current-state-by-design |
Caching keys keep their existing names (values change source); BQ-backed entries move to the bq: prefix convention only where new keys are created.