Lantern

Appearance

Analytics configurations page and the live retention panel - Manual Test Plan ​

FieldValue
Branch / PRfeat/admin-and-merchant-portals
EnvironmentLocal dev: admin portal :3001 proxying to analytics-api, which reads LIVE lantern-app-dev BigQuery
Build flags / configNone. The page is not flag-gated
Build (commit)fill in at run time
Tester
Date
Designdocs/projects/analytics-config-and-reports/design.md
Issue#987

Summary ​

#ScenarioResultNotes
1The page renders with retention on it[ ] pass [ ] fail [x] blockedPortal proxies to the DEPLOYED api, see Notes
2The retention table DISAGREES with the old hard-coded panel, in three named places[ ] pass [ ] fail [x] blockedThe falsifying scenario. Passes at the model layer, see Notes
3The two expirations still AGREE with the old panel[ ] pass [ ] fail [x] blockedThe control. Passes at the model layer
4require_partition_filter is shown for analytics.events[ ] pass [ ] fail [x] blockedSame block
5The page states its own scope[ ] pass [ ] fail [ ] blocked
6ACTIVATION: the route is reachable with no manual setup[x] pass [ ] fail [ ] blockedAgent run 2026-08-28
7ACTIVATION: the nav entry is present under Analytics, not Settings[x] pass [ ] fail [ ] blockedAgent run 2026-08-28
8The old bigquery/retention path redirects[x] pass [ ] fail [ ] blockedAgent run 2026-08-28
9The BigQuery page no longer offers a Retention tab[ ] pass [ ] fail [ ] blocked
10The endpoint failing does not render as empty or as zero[ ] pass [ ] fail [ ] blocked
11Non-admin cannot reach the page[ ] pass [ ] fail [ ] blocked
12REGRESSION: Query Console and the BigQuery export tab still work[ ] pass [ ] fail [ ] blocked

Setup / preconditions ​

  • Admin portal running on :3001 and the analytics-api it proxies to. Do not restart the operator's servers; check what is already listening first.
  • Sign in as an admin. Both portal signin routes share ONE rate limit bucket of 10 per 15 minutes per IP, shared with the operator's own session, so reuse a session rather than signing in repeatedly.
  • The analytics-api service account needs roles/bigquery.metadataViewer. Without it GET /analytics/admin/bq-schema returns 403 with a hint naming that role; scenario 10 covers that path deliberately.
  • Ground truth for the retention values, to check the UI against. Run in BigQuery, or use the copy recorded in the design doc:
sql
SELECT t.table_name, o.option_value AS partition_expiration_days,
       ARRAY_TO_STRING(ARRAY_AGG(c.column_name ORDER BY c.clustering_ordinal_position), ', ') AS clustering
FROM `lantern-app-dev.analytics.INFORMATION_SCHEMA.TABLES` t
LEFT JOIN `lantern-app-dev.analytics.INFORMATION_SCHEMA.TABLE_OPTIONS` o
  ON o.table_name = t.table_name AND o.option_name = 'partition_expiration_days'
LEFT JOIN `lantern-app-dev.analytics.INFORMATION_SCHEMA.COLUMNS` c
  ON c.table_name = t.table_name AND c.clustering_ordinal_position IS NOT NULL
WHERE t.table_type = 'BASE TABLE' GROUP BY 1, 2 ORDER BY 1
  • Nothing here writes. Every scenario is read-only against BigQuery metadata. No destructive steps in this plan.

Scenarios ​

1. The page renders with retention on it ​

  • Goal: The configurations page exists and carries the retention panel.
  • Preconditions: Signed in as admin.
  • Steps:
    1. Navigate to the Analytics configurations page.
    2. Wait for the retention table to populate.
  • Expected: A page with a retention section listing the analytics base tables, each with its partition column, expiration and clustering. No console errors.
  • Verify (data/console): Network tab shows a request to /analytics/admin/bq-schema (or its successor) returning 200.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

2. The retention table DISAGREES with the old hard-coded panel, in three named places ​

  • Goal: Prove the panel is READING BigQuery rather than rendering a constant. This is the falsifying scenario in this plan.
  • Why it can lose: the old panel's values are known and three of them are wrong. If the new panel shows the OLD values, it is not reading anything, and the change has not been made regardless of how the page looks.
  • The prediction, stated before the run:
RowOld hard-coded panel saidLive panel MUST say
analytics.events clusteringevent_nameevent_name, user_id
analytics.event_counts_daily clusteringnot setevent_name
analytics.ad_delivery_dailyabsent from the table entirelypresent, partitioned on day, no expiration
  • Steps:
    1. Open the configurations page.
    2. Read the clustering cell for analytics.events.
    3. Read the clustering cell for analytics.event_counts_daily.
    4. Check whether analytics.ad_delivery_daily appears as a row at all.
  • Expected: All three match the "Live panel MUST say" column.
  • Verify (data/console): Compare against the setup query's output, not against this document, in case the tables changed since it was written.
  • If any of the three still shows the OLD value: the panel is still hard-coded, or the reader is not reaching the metadata. Fail, and do not pass this on the strength of the page looking right.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

3. The two expirations still AGREE with the old panel ​

  • Goal: The control for scenario 2. Our account of the cause says the old panel was wrong about clustering and RIGHT about expirations, so these two must not move.
  • Prediction, opposite to scenario 2: these values do NOT change.
  • Steps:
    1. Read the expiration cell for analytics.events.
    2. Read the expiration cell for analytics.event_counts_daily.
  • Expected: analytics.events shows 90 days. analytics.event_counts_daily shows no expiration.
  • What a surprise means: if either of these changed, the diagnosis in the design doc is incomplete. It would mean the old panel was not simply stale on clustering, and the reader's mapping of BigQuery fields to columns needs re-checking before any of scenario 2's result can be trusted.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

4. require_partition_filter is shown for analytics.events ​

  • Goal: The panel surfaces the thing that makes a query FAIL rather than run slowly.
  • Steps:
    1. Find the analytics.events row.
    2. Look for an indication that a partition filter is required.
  • Expected: The row shows that analytics.events requires a partition filter. The other two tables do not.
  • Verify: The setup query's require_partition_filter column: true for events, false for the other two.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

5. The page states its own scope ​

  • Goal: A future addition has a test to pass rather than a gap to fill.
  • Steps:
    1. Read the page's scope line.
  • Expected: The page says in plain words that it holds settings governing how analytics data is STORED and KEPT, and that things changing what is measured or what is shown belong elsewhere. It does NOT show empty placeholder settings, "coming soon" rows, or disabled controls for settings that do not exist.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

6. ACTIVATION: the route is reachable with no manual setup ​

  • Goal: Prove the route is REGISTERED, not just that the component works. A component that renders in Storybook proves nothing about the shipped app.
  • Preconditions: A fresh browser profile, or a hard reload with cache disabled. No manual navigation trick, no direct component mount.
  • Steps:
    1. From a clean session, sign in.
    2. Reach the configurations page by clicking through the nav only, never by typing the URL.
  • Expected: The page is reachable by clicking. It renders, and does not fall through to the Dashboard catch-all.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

7. ACTIVATION: the nav entry is present under Analytics, not Settings ​

  • Goal: The placement decision actually shipped. Configuration lives with the thing it configures.
  • Steps:
    1. Expand the Analytics section in the sidebar.
    2. Expand Settings.
  • Expected: The configurations entry appears under Analytics. It does not appear under Settings.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

8. The old bigquery/retention path redirects ​

  • Goal: A saved link or bookmark still lands somewhere sensible. This is the repo's house convention on its fourth application.
  • Steps:
    1. Type /admin/analytics/bigquery/retention into the address bar.
    2. Press enter.
  • Expected: The browser lands on the new configurations page. It does NOT land on the Dashboard, and does NOT show a blank screen.
  • Verify: The URL bar ends up on the new path (a <Navigate replace>, so Back does not bounce).
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

9. The BigQuery page no longer offers a Retention tab ​

  • Goal: Retention MOVED rather than being duplicated. Two copies is the outcome this project exists to reduce.
  • Steps:
    1. Open the BigQuery page.
    2. Read its tabs.
  • Expected: No Retention tab. The remaining tabs still work.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

10. The endpoint failing does not render as empty or as zero ​

  • Goal: A service that did not answer must say so. It must never render as "no retention policies", which reads as a fact.
  • Steps:
    1. Simulate the failure. Either block the request in devtools, or point the portal at an analytics-api whose service account lacks roles/bigquery.metadataViewer.
    2. Reload the configurations page.
  • Expected: The page says the retention data could not be read, and ideally why. It does NOT render an empty table, a "0 tables" count, or a silently absent section.
  • Verify: A 403 from the endpoint carries code: PERMISSION_DENIED and a hint naming the role. Check that reaches the user in some usable form.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

11. Non-admin cannot reach the page ​

  • Goal: The page inherits the admin gate rather than relying on the nav not showing it.
  • Steps:
    1. Sign in as a non-admin (a merchant account).
    2. Type the configurations page URL directly.
  • Expected: Access denied, or a redirect away. The retention data must not render, and the network tab must not show a successful metadata response.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

12. REGRESSION: Query Console and the BigQuery export tab still work ​

  • Goal: The surfaces this change did NOT intend to touch are untouched.
  • Steps:
    1. Open Query Console. Run a small query.
    2. Open the BigQuery export tab. Confirm coverage renders.
    3. Type /admin/analytics/bigquery/console and confirm it still redirects to Query Console.
  • Expected: All three behave exactly as before. The console's own schema browser still lists datasets and tables.
  • Note: the schema browser shares the bq-schema endpoint with this work. If that endpoint is changed to carry descriptions, this scenario is what catches a break in its existing consumer.
Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:   (screenshot / console line / doc id)
Follow-up:

Notes ​

  • Which scenario can prove us wrong: number 2, with number 3 as its control. If the retention panel shows the OLD clustering values, it is not reading BigQuery, no matter how the page looks. If the CONTROL moves (the expirations change), the diagnosis in the design doc is incomplete and reopens.
  • Scenarios 6, 7 and 8 are blocked until the route is registered in AdminShell.jsx, which is sequenced against task 1 and is deliberately not done yet.
  • A post-deploy pass is still owed on scenarios 6, 7, 8 and 10 against the DEPLOYED build. A pre-merge pass proves the code works locally; it does not prove the shipped artifact carries the config, and the service account permission in scenario 10 is exactly the kind of thing that differs between local and deployed.

Agent run, 2026-08-28 ​

Driven headless against the operator's running portal on :3001. Read only, no servers restarted.

What passed live ​

  • Scenario 8, the redirect. /admin/analytics/bigquery/retention lands on /admin/analytics/config. No fall-through to the Dashboard catch-all.
  • Scenario 7, the nav entry. "Configurations" is present under Analytics.
  • Scenario 6, the route. The page renders under the admin shell with the right title and zero page errors.

What is BLOCKED, and the cause is settled rather than guessed ​

Scenarios 1 to 4 need the retention table, which is not rendering, and the page is CORRECT to not render it.

The cause, proven in one command: ANALYTICS_API_ORIGIN=https://analytics-api-531553779372.us-central1.run.app. The admin portal proxies /api/analytics to the DEPLOYED Cloud Run service, not to the local one on :8082. The deployed service predates the metadata passthrough, so it returns the old payload shape, and the page shows its not-wired state exactly as designed.

The response body confirms it rather than the config alone: a 200 from /api/analytics/admin/bq-schema whose tables carry {id, type, columns:[{name,type,mode}]} and no timePartitioning, clustering or description.

Three candidate causes were possible and two are ruled out:

CandidateRuled out how
The page or the route is brokenIt renders, the redirect works, no page errors
The local analytics-api has not reloaded (WSL2 node --watch)Not the service being called. The proxy goes to Cloud Run
The portal proxies to the deployed serviceThis one. Confirmed by the env value and the response shape

To unblock scenarios 1 to 4, either the change deploys, or ANALYTICS_API_ORIGIN is pointed at http://localhost:8082 and the vite server restarted (it reads env at startup). Both are the operator's call; no server was restarted for this run.

Superseded 2026-08-30: the localhost origin is now the DEFAULT. .env.local.example ships ANALYTICS_API_ORIGIN=http://localhost:8082 and VENUE_API_ORIGIN=http://localhost:8080 (#1002), because two of six origins pointing at Cloud Run made a branch backend change invisible locally and cost three separate investigations. A dev server started after that change proxies to the LOCAL analytics-api, so this block clears on a portal restart rather than on a deploy. The Cloud Run origin is now the opt-in, for a service you are not running locally.

The falsifying scenario passes at the layer that CAN be tested today ​

Scenario 2 is the one that can prove the diagnosis wrong, and it is asserted in retentionModel.test.js against a fixture read out of the real warehouse: the rows must show events clustered on event_name, user_id, event_counts_daily clustered on event_name, and ad_delivery_daily present. Its control, scenario 3, asserts the two expirations do NOT move. Both pass.

That is not the same as passing end to end, and it is recorded as blocked rather than passed for exactly that reason.

One defect found and fixed during the run ​

The loading state and the empty state both rendered .acfg-muted, so a probe reported "no base tables" while the panel was still loading. Two different facts wearing one class. Each state now carries its own class as well, which is what made the run above able to tell notwired from empty.

The post-deploy list, for tonight ​

Written now rather than reconstructed when the deploy lands. This is the whole of it, and the short version is that ONE thing is deploy-gated and it unblocks everything else.

What IS deploy-gated: does the deployed endpoint carry the new fields? ​

Everything blocked today traces to one fact. ANALYTICS_API_ORIGIN points the admin portal at the deployed Cloud Run analytics-api, and until tonight's merge deploys it, that service predates the metadata passthrough.

The single check that settles it, signed in as admin on the deployed portal:

GET /api/analytics/admin/bq-schema

Look at any table in the response for four keys: description, timePartitioning, clustering, requirePartitionFilter. If they are present, everything below unblocks at once. If they are absent, the deploy did not carry the change and nothing else in this list is worth running.

Then re-run scenarios 1 through 4, which are the retention table. They should flip from the not-wired state to real rows, and scenario 2 is the falsifying one: the values must DISAGREE with the old hard-coded panel in exactly three places.

One specific risk worth checking rather than assuming ​

Dataset descriptions may come back null in production while they are non-null locally, and it would not look like an error.

The passthrough makes a second call per dataset, ds.getMetadata(), which needs bigquery.datasets.get. roles/bigquery.metadataViewer includes it, but the deployed service account's exact grant has not been checked. A failure there is swallowed to null on purpose, so one unreadable dataset loses its description rather than taking the whole schema request down.

That graceful degradation is right, and it means a permissions gap presents as "no dataset descriptions" rather than as a 403. So:

  • If table and column descriptions arrive but every DATASET description is null, the cause is the service account, not the code. Locally all nine datasets except two return one.
  • Ground truth to compare against: analytics should read "Lantern analytics events - unified pipeline for client and server-side tracking".

What is NOT deploy-gated, and it is more than you might expect ​

The schema viewer and the Report Builder have no live checks tonight at all, because neither is routed. That was deliberate: routing them today would have shipped two pages showing a not-wired state for one evening.

They are fully verified at the level they exist at: 15 and 14 model tests against a real warehouse capture, twelve stories between them, zero axe violations, zero native select elements. Their first live verification comes with whatever change routes them, and that change is small and verifiable in a single pass once the endpoint carries the fields.

Also not deploy-gated: the three migrated Dashboards placeholders (verified live today on all three routes, real dates, 123 days), the configurations route and its redirect (scenarios 6, 7 and 8, passed today), and the eslint and docs-format changes, which are CI-gated rather than deploy-gated.

The order to run it in ​

  1. The endpoint check above. If it fails, stop; nothing else is meaningful.
  2. Scenarios 1 to 4, the retention table, with scenario 2 as the falsifier and 3 as its control.
  3. Scenario 10, the failure path, which is worth doing on the deployed build specifically because the service account differs from local.
  4. Scenarios 6, 7, 8 again on the deployed build, since a pre-merge pass proves the code works, not that the shipped artifact carries the config.

Built with VitePress

Copyright © 2026 Lantern