Application monitoring: what to review

Field	Value
Branch	feat/admin-and-merchant-portals, draft PR #976: feat(admin): portal navigation, financials and monitoring, plus the agent-process work behind them, not merged
Project	README.md
Issue	#164: Infra: Application Monitoring & Observability
Servers you need	Admin portal on http://localhost:3001 (npm run admin:dev), plus a local analytics-api on 8082 (npm run analytics-api:dev)
Local origins	Required, and this project is the clearest demonstration of why. See the warning below

Why is this the clearest case for the local origins?

System Health Uptime shows 4 endpoints against the deployed analytics-api and 9 against a local one.

• One browser, one session, only the origin changed. If you review this against the default .env.local you will see the old four-endpoint tab, three of them red, and conclude the work did not land.
• The chain is SystemHealth.jsx to getSystemHealth() to VITE_ANALYTICS_API_URL=/api to the Vite proxy's /api/analytics to ANALYTICS_API_ORIGIN, which points at Cloud Run.
• There is also a 5-minute server-side cache on admin:systemHealth behind that, so a change can take a beat even once the origin is right.
• Tracked as #1002: fix(dev-env): local dev proxies venues and analytics to Cloud Run, so branch backend changes are invisible locally.

What changed here?

The Uptime tab was measuring almost nothing, and three of its four checks could never pass.

• It checked four hardcoded URLs. None of the seven Cloud Run services was covered, so the tab could report "operational" while saying nothing at all about the backend.
• admin.ourlantern.app and storybook.ourlantern.app do not resolve (curl returns 000). Two of the four checks were probing hostnames that were never provisioned and reporting DOWN for it.
• The Discord bot has no HTTP listener anywhere. It is a gateway client, so the probe could never pass. Removed rather than left red, because a check that cannot succeed teaches everyone to ignore the tab.
• The API half now derives from packages/shared/services, so a service is monitored the day it is registered.

The cold-start allowance was set from measurement, not from a guess.

• Measured 2.6s to 9.2s cold and 0.3s to 0.5s warm. An 8s bar flagged assistant-api on every cold call, so the bar is 12s: above the worst cold start with headroom, so tripping it means genuinely slow rather than merely cold.

The Errors tab was reading a collection nothing has ever written to.

• clientErrors holds zero documents in live dev, and nothing anywhere writes to it. The panel said "No errors logged" unconditionally, and its empty state promised errors are automatically captured, which was never true.
• It now reads Cloud Logging for severity>=ERROR, because every Cloud Run service, Cloud Function and Scheduler job already writes failures there. That is the store that already has everything, rather than a second error store nobody would remember to instrument.
• The client half is kept and its empty state now says what it means: not instrumented, rather than none happened.

It immediately surfaced a real bug: the nightly aggregation job had 404ed for days.

• 34 server errors in 7 days, 4 in the last 24 hours, across 5 sources, on the tab that reported none. Twenty of them were analytics-daily-aggregation, a Cloud Scheduler job hitting a route retired in the #616: analytics: realtime from Firestore, historical from BigQuery; drop the redundant analytics_events mirror cleanup, four retries a night at 02:00 UTC.
• The job was deleted on your explicit permission, after verifying the route is genuinely gone and backing up the full job definition. The other 13 jobs are untouched.
• The roles/logging.viewer grant was not needed and was not made: analytics-api already runs as a service account holding roles/editor, which includes logging.logEntries.list.

Where do I look, and what am I looking for?

System Health, Uptime tab http://localhost:3001/admin/system, then Uptime

• Needs the local origins.
• Nine endpoints, not four. Two frontends plus the seven registered APIs. All should be healthy, average around 800ms warm.
• Nothing should be reporting DOWN for a hostname that was never provisioned, and the Discord bot should be absent entirely rather than red.
• A capture of the verified run is at ../merchant-and-venue-dashboards/harness/uptime-branch-verified.png.

System Health, Errors tab http://localhost:3001/admin/system, then Errors

• Needs the local origins. Against a local analytics-api this panel degrades: Cloud Logging answers 403 because the local service account lacks roles/logging.viewer, and the section says so by name. That is the service behaving as written, not a failure.
• Against the deployed origin the Server Errors section reads real Cloud Logging data, grouped by source with a recent feed.
• The client half should say "nothing currently writes to the clientErrors collection, so treat this as not instrumented rather than none happened". It must not say "No errors logged".
• A capture is at ../merchant-and-venue-dashboards/harness/errors-tab-server-errors.png.

System Health sits under Development in the sidebar now, not Operations.

• It is a developer surface, so that is where it belongs. Confirm the section order reads Main, Platform, Operations, Development, Settings.

What is deliberately NOT done?

Nothing captures client errors at all. This is the largest remaining hole and it is the next task.

• No window.onerror, no unhandledrejection handler, no Sentry in either app.
• The admin portal has one ErrorBoundary whose componentDidCatch only calls console.error. The web PWA, the app users actually run, has no boundary at all.
• So a user hitting a crash produces nothing anyone can see.

Cloud Error Reporting is already running on lantern-app-dev and has never been surfaced.

• It groups errors by signature and carries resolutionStatus (OPEN, ACKNOWLEDGED, RESOLVED, MUTED), first and last seen, count and affected services. Five groups are currently OPEN, the oldest first seen 2026-01-30.
• It answers "is this error resolved", which Cloud Logging cannot. It is complementary, not a replacement: it shows 5 groups where Logging showed 34 entries, because the Scheduler 404s are not exceptions.
• A resolved group that recurs is the signal that matters, and it comes free: lastSeenTime moving after a group was marked RESOLVED means the fix did not hold.
• Surfacing it is a sequenced task in planning.md. Not built.

No alerting exists.

• The admin Dashboard rail is shaped to take a threshold row and nothing feeds it. Your side quest for a dedicated alerts page, so anomaly signals are centralized rather than scattered, is recorded and not scoped.

Upstream platform status is a separate side quest, not started.

• Monitoring whether GCP or Cloudflare are themselves down, distinct from the uptime check, which only proves our endpoints answer.

The Discord bot is now unmonitored.

• Monitoring it for real needs either an HTTP health surface on the bot or a gateway-side check. Named here rather than left as a red row nobody trusts.

What open findings touch this project?

Cross-referenced from ../merchant-and-venue-dashboards/branch-review-0828.md. Both of the review's mid-tier findings land here, and both are on surfaces this project just built.

The uptime rollup reports operational when every endpoint has timed out. systemHealth.service.js:1110

• The rollup sets status only from summary.down. A timeout increments summary.degraded, so all nine endpoints timing out still reports operational.
• Open, not fixed. It is the exact failure mode this project exists to prevent, one layer up from the one it fixed.

The Errors tab states a capped number as an exact count. systemHealth.service.js:908

• nextPageToken is discarded and truncation is inferred from entries.length >= 200.
• Open, not fixed. The 24h card was separately fixed to stop under-reporting while the 7d card beside it admitted its cap; this is the remaining half.

What issue does each piece close?

Piece	Issue	State
Uptime, server errors, and everything else in this area	#164: Infra: Application Monitoring & Observability	Open, deliverable area, partially built
The nav rearrange this work came out of	#320: Phase 3: Venue analytics, aggregate interest and vibe insights for merchants	Recorded in ../merchant-and-venue-dashboards/README.md
The local origin flip that makes Uptime readable here	#1002: fix(dev-env): local dev proxies venues and analytics to Cloud Run, so branch backend changes are invisible locally	Open, another lane