Post-deploy window, 2026-08-18 evening - Runbook
Three merges deploy tonight and each owes a deployed-build verification. This page exists so the 18:30 runner executes rather than decides. Every pass below has its command or click path and its expected result.
Nothing here is a substitute for the per-change test plans, which carry the full scenario sets and the fill-in blocks. This is the deployed subset, ordered.
| Pass | Change | Plan | Login needed |
|---|---|---|---|
| A | Billing donut rows (#913 / PR #918) | 913-billing-donut-row | admin portal |
| B | Billing Reports reconciliation (#912) and Overview invoice-only vendors (#915), both in PR #916 | 912-billing-reports-reconciliation, 915-overview-invoice-only-vendor | admin portal |
| C | Milestone classifier (#917) | see that PR's plan | none (API key only) |
0. Pre-flight, and do not skip it
Confirm the deployed artifact actually changed before running anything. On 2026-08-17 a merge landed and the deploy never fired because CI hung; a pass run then would have measured the previous bundle and passed.
curl -s https://dev.ourlantern.app/version.jsonRecord buildId and buildTime. buildTime must be after tonight's merges. If it is not, the deploy has not landed: stop, say so, and do not run passes A or B. Pass C does not depend on a deploy.
Today's pre-merge baseline for comparison: buildId msyto7tz, built 2026-08-18T15:33:24Z.
Logins, and one trap
Both admin passes use ~/.lantern-agent-probe.env (0600).
The admin sign-in authenticates on the LEGACY Firebase tier, not the admin-password tier. App Check refuses tiers 1 and 2 in a headless browser, so the portal falls through. This is expected, is not a fault of tonight's changes, and is filed as #924. Do not spend the window rediscovering it.
Two more things that waste time if unknown, both filed as #922:
- The sidebar items and dashboard cards are divs with
onClick, sogetByRole('button')matches nothing. Click by visible text. - Do not drive tabs by URL hash. The admin router owns the hash, so
#reportsleaves Billing entirely and lands on the dashboard.
Pass A: billing donut rows (#913)
One command, no click path.
node tooling/e2e/billing-donuts.e2e.mjsMeasures both donut rows at 412 / 1024 / 1280 / 1440 / 1920 and fails if any card is clipped by .admin-main's overflow-x: hidden.
Expected: DONUT PASS GREEN, and in the printed table:
| Width | Overview | Reports |
|---|---|---|
| 412 | 1 col, nothing clipped | 1 col, nothing clipped |
| 1024 | 2 col | 2 col, nothing clipped |
| 1280 | 3 col | 3 col |
| 1440 | 3 col | 3 col |
| 1920 | 3 col | 3 col |
The load-bearing row is Reports at 1024. On the pre-fix build that width rendered three columns and clipped the third donut entirely. If 1024 shows CLIPPED, the fix did not deploy.
On failure the script screenshots to /tmp/donuts-*.png.
This script is already proven to FAIL on the pre-fix build, run against msyto7tz at 2026-08-18 before the fix deployed:
Reports donuts
1024 | 3 | 254 | CLIPPED 128px
FAIL Reports donuts @ 1024px: nothing clipped :: clipped 128px
...
10 checks, 1 failed
DONUT PASS FAILEDNine other checks passed in the same run, so it is not failing indiscriminately. Tonight it should go green, and that flip is the evidence. A green from a script never seen to go red would only prove it ran.
Pass B: billing Reports reconciliation (#912)
Click path, admin portal › Billing › Reports. Full scenarios are in that change's plan; this is the deployed subset in execution order.
B1. July 2026 headline (plan scenario 1). Period Monthly, step to July 2026.
- Summary Total = $269.68 (not $79.97)
- Anthropic present at $200.00 (74%)
- GitHub reads $48.85 (its invoiced figure, not the $59.15 line-item estimate)
- Vendor breakdown lists 4 vendors, subtitle says 4, rows sum to the Total
B2. No self-disagreement (scenario 2). Same screen.
- Summary Total, the By-Vendor donut centre, and the sum of vendor rows all agree
- Railway reads $5.00, not $6.36
B3. Expand the GitHub row (scenario 4).
- Row spend stays the invoiced $48.85
- Service detail shows line-item usage (~$59.15) with a caption saying it may differ from the invoiced spend above
- Shares stay within 100%
B4. Current month labelling (scenario 5). Set period to the current month.
- Hint reads "month to date", NOT "estimate"
- Avg/day divides by elapsed days
- vs-Previous compares against the previous month prorated to elapsed days
B5. Sub-month grain (scenario 8). Pick "Last 7 days" or a custom range.
- Summary FYI reads " Line-item totals, net of credits"*
- Tooltip says invoice-only vendors and reconciled totals are not reflected
- The copy does not claim reconciliation
B6. No fake collapse (scenario 9). On the current partial month.
- vs-Previous shows no large fake negative (not a raw -85% against July's full $269.68)
B7. Overview's current month now includes the invoice-only vendor (#915, which ships inside the same PR). Billing › Overview, read "Monthly / current month" and the per-vendor figures.
- Anthropic appears at its posted invoice for the live month (~$200), not $0
- The Monthly total includes it
- Metered vendors are unchanged: GCP, GitHub and Cloudflare still show fact month-to-date and must not move
Control, captured today on the pre-fix build msyto7tz at 16:03 PT. The Overview Monthly donut read:
| Vendor | Amount | Share |
|---|---|---|
| GCP/Firebase | $32.56 | 51% |
| GitHub | $26.52 | 41% |
| Railway | $5.00 | 8% |
| centre total | $64.08 |
Anthropic absent. So tonight's B7 is a flip, not a first impression: the same screen should gain an Anthropic row and a larger total, while those three figures stay put.
Two expected results that look like bugs
Reports and Overview can differ for the current month, by design (#912 scenario 7). They read different tables; both are month-to-date. Do not file this. Note the difference CHANGES tonight, because #915 alters Overview's side of it; #912's scenario 7 is the reference for what the remaining difference means.
Reports' current month may still exclude Anthropic's live-month invoice. #915 fixes the OVERVIEW path (currentMonthByVendor); the commit and its plan both scope it to Overview. Do not assume the same change applies to Reports. #912 scenario 5 is the reference for Reports' current-month behaviour, and if Reports still excludes it that is the documented state, not tonight's bug.
This entry was wrong in the first draft of this runbook, which said Anthropic's exclusion "is not something tonight's change fixes". fe07699c is on the #916 branch and fixes exactly that for Overview. Corrected after reading the commit rather than the summary, and the surface it names matters: Overview, not Reports.
Pass C: milestone classifier (#917)
node --env-file=.env.local tooling/scripts/classify-milestone-dryrun.mjs --per-stage=6Needs ANTHROPIC_API_KEY in .env.local and normal gh auth. No portal login, so #924 does not apply here.
What it proves (devtools' words, kept verbatim):
the real classifier code path. The dry run imports the same
classify()module the workflow invokes, makes real API calls, and scores the answers against issues the operator sorted by hand. A regression in the prompt, the allowlist validation, or the decline logic shows up here.
What it does NOT prove:
that the workflow works. It never runs the workflow's shell, never assigns a milestone, and never posts the audit-trail comment. Those three are where the only bug found so far actually lived.
Read that second half before reporting this pass as coverage. The one bug caught so far was in the workflow's shell, not the classifier: reasoning was set inside a $(...) subshell so it never reached the caller, and every audit-trail comment would have shipped with an empty reason. This dry run passes cleanly through that, and so did YAML validation, actionlint and review. A check on the classifier is not a check on the workflow.
The live-fire is a watch-item, not a manufactured event
The classifier fires only on a newly opened issue, and nobody creates test issues on her tracker. So:
The live-fire check is a watch-item on the next real issue somebody files: did it get a plausible stage plus a comment saying a bot assigned it, or was it left unmilestoned with a warning in the run log? Before reporting a wrong stage as a bug, check the known miss class first: security or privacy work implemented as tooling gets filed Prototype where she sorts it Alpha.
That last sentence prevents a documented, measured, expected miss being investigated from scratch as a new defect.
Recording results
Put results in each change's own plan, not here, so the record lives with the change. This page is the order of operations and can be deleted or superseded once the window closes.
If a pass is red, capture the artifact (screenshot, printed table, run log) before re-running. A re-run that goes green without an explanation of the first red is not a pass.