Lantern

Appearance

Admin portal sign-in tiers (#924, #904) - Manual Test Plan ​

FieldValue
Branch / PRfix/924-admin-auth-pair
EnvironmentDeployed dev: https://admin.dev.ourlantern.app against auth-api on lantern-app-dev, plus a local admin build on a non-default port for the branch-only client changes
Build flags / confignone new. App Check is ENFORCED on deployed dev (K_SERVICE is set on Cloud Run, so the local bypass never applies)
Build (commit)live runs captured on e7becd80; see the run-provenance note below, which says which arms were re-derived afterwards
Testeragent (headless Playwright) unless noted
Date2026-08-20

The cell-by-cell reasoning behind these scenarios, including the cells deliberately left unchanged, is in the scenario matrix. Scenario numbers below map to the matrix cell ids in brackets.

Summary ​

#ScenarioResultNotes
1Agent Probe credential set through the real reset flow [C15][x] pass [ ] fail [ ] blockedlanded in private/auth, reset-required cleared, audit row wasReset:true
2Tier 1 signs the probe in on the admin portal password [C1][x] pass [ ] fail [ ] blockedfirst time tier 1 has EVER been exercised headlessly
3Wrong password does NOT fall through to the legacy tier [C2][x] pass [ ] fail [ ] blockedA/B: deployed build signed in on legacy, branch build refused
4App Check failure is refused, not downgraded [C13][x] pass [ ] fail [ ] blockedA/B: deployed build signed in unattested, branch build refused
5Forced password reset cannot be sidestepped [C5][x] pass [ ] fail [ ] blockedA/B: deployed showed "Incorrect password", branch showed the reset instruction
6A 429 holds instead of falling through [C4][x] pass [ ] fail [ ] blockedrun live; the 429 observed was the per-IP limiter, not the lockout. Surfaced a real message bug, fixed
7Banned admin with no portal password is refused [C7][x] pass [ ] fail [ ] blockedemulator only, deliberately: not banning a real dev admin
8Migration window still open for a genuinely password-less admin [C6][x] pass [ ] fail [ ] blockedthe regression guard. 428 then straight to legacy, tier 2 skipped
9Merchant sign-in still works (the one fallthrough that must survive) [C9][x] pass [ ] fail [ ] blockedtier 1 401, tier 2 200. The role-disambiguation hop survives
10Legacy tier is countable from the audit stream [I6][ ] pass [ ] fail [x] blockedserver change is not on deployed dev yet; emulator-verified, live check is scenario 11
11Post-deploy: the tier discipline shipped, on a clean profile[ ] pass [ ] fail [x] blockedruns after merge and deploy

Setup / preconditions ​

  • Accounts. Agent Probe test-admin, credentials in ~/.lantern-agent-probe.env (0600). Merchant test login in ~/.lantern-test-accounts.env (0600). Never paste either into a command line; source the file.
  • App Check. Headless runs inject the registered debug token via self.FIREBASE_APPCHECK_DEBUG_TOKEN in a Playwright addInitScript before page load. The value is APPCHECK_DEBUG_TOKEN in ~/.lantern-test-accounts.env. Without it, deployed dev returns 401 APP_CHECK_REQUIRED on tiers 1 and 2.
  • Branch-only client changes. Scenarios 3 to 9 exercise portalSignIn.js, which is not on deployed dev until this PR merges. Run them against a local apps/admin build on a non-default port, pointed at the deployed dev auth-api. Scenario 11 is the post-merge repeat against the deployed build.
  • Server-side scenarios (7, 10) exercise auth-api and are covered by the emulator suite npm run test:admin-signin:emulator; the manual entries here record the live confirmation where one is possible without mutating a real account.
  • Data to verify in. Firestore project lantern-app-dev: adminProfiles/{uid} (and its private/auth subdoc), adminActions (the audit stream), adminPasswordResetTokens.
  • Destructive steps are flagged inline. Scenario 6 locks a real account for the lockout window; scenario 7 would require banning a real admin and is therefore run on the emulator only, never live.

Scenarios ​

1. Agent Probe credential set through the real reset flow [C15, #904] ​

  • Goal: the probe gets an admin portal password by the same route a real admin takes, so the fixture tests something a real admin could reach.
  • Preconditions: probe account has no private/auth subdoc and adminPasswordResetRequired: true.
  • Steps:
    1. POST /auth/admin/password/reset for the probe email (App Check token required).
    2. Read the generated token from adminPasswordResetTokens with the Admin SDK against lantern-app-dev.
    3. POST /auth/admin/password with that resetToken and a freshly generated password.
    4. Write the password into ~/.lantern-agent-probe.env (0600) without echoing it.
  • Expected: step 3 returns 200 Admin password set successfully.
  • Verify (data/console): adminProfiles/{uid}/private/auth now exists; adminProfiles/{uid}.adminPasswordResetRequired is false; adminPasswordSetAt is stamped; an adminActions row with action: setAdminPassword, wasReset: true. LIVE write to a real dev account, deliberately, per #904.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    All four steps returned as expected. Reset requested through the portal's own Forgot password flow (App Check debug token injected), token read from `adminPasswordResetTokens` with the Admin SDK on dev, password set on the real SetAdminPassword screen.
Evidence:  uid `vyRySTF7cNUYv0CfPjK1l2ISjwy2`. Network: 200 POST /auth/admin/password/reset, 200 GET /auth/admin/password/reset/:token, 200 POST /auth/admin/password. After: `private/auth` exists with `[passwordHash, passwordSalt, updatedAt]`, `adminPasswordResetRequired: false`, `adminPasswordSetAt` stamped, no inline `adminPasswordHash` (so it took the #875 write path). adminActions `MI0VvcSezwGN868NDyMM`, `wasReset: true`.
Follow-up: Answers the open question in #904: the account NEVER finished setup, the env file was not stale. `users/{uid}` has no `salt`, so this admin has no Lantern account, which also means `AdminMigrationBanner` would never have prompted it. See the matrix assumption check.

2. Tier 1 signs the probe in on the admin portal password [C1] ​

  • Goal: prove tier 1 actually authenticates, for the first time. Every agent-driven admin sign-in to date landed on tier 3 without anyone noticing.
  • Preconditions: scenario 1 passed.
  • Steps:
    1. Launch headless chromium with the App Check debug token injected.
    2. Load the admin portal login screen, enter the probe email and its new portal password.
    3. Capture every network request the page makes during sign-in.
  • Expected: the portal loads signed in as the probe.
  • Verify (data/console): POST /auth/admin/signin returned 200, and signInWithPassword (the Google Identity Toolkit legacy endpoint) was NOT called at all. That absence is the assertion; a successful login alone does not distinguish tier 1 from tier 3. Also: a new adminActions row with action: adminLogin, success: true, tier: 'adminPassword'.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Signed in on the deployed build. Tier 1 answered 200 and nothing else was called.
Evidence:  `{"signedIn": true, "tiersCalled": ["200 tier: 1 admin"], "legacyTierUsed": false}`. The absence of the Identity Toolkit call is the assertion. Repeated on the branch build with the same result.
           NOT CHECKED, and the Verify line above overstated it: the `tier: 'adminPassword'` audit row is a SERVER change and deployed dev still runs the previous auth-api build, so no live run could have seen it. Covered on the emulator; the live check belongs to scenario 11. Flagged by the review round, which was right that a pass mark against an unperformed Verify is the worst kind, because it reads as checked.
Follow-up: Closes #924 point 3. Tier 1 had never been exercised by an agent before this run.

3. Wrong password does NOT fall through to the legacy tier [C2] ​

  • Goal: the defect at the centre of #924. A portal password must retire the legacy password it replaced.
  • Preconditions: the probe has a portal password (scenario 1) and, ideally, a still-valid legacy Firebase password.
  • Steps:
    1. Sign in as the probe with a password that is NOT the portal password.
    2. Capture the network requests.
  • Expected: "Incorrect email or password." and no session.
  • Verify (data/console): POST /auth/admin/signin returned 401, POST /auth/merchant/signin returned 401, and no call to the Identity Toolkit signInWithPassword endpoint. Before this branch, that third call happened and succeeded whenever the typed password was the legacy one.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    The defect reproduced on the deployed build and is refused on the branch build, using the SAME account and the SAME credential. The probe now has a portal password, and its legacy Firebase password should no longer sign it into the portal.
Evidence:  Deployed (shipped client): `["401 tier: 1 admin", "401 tier: 2 merchant", "200 tier: 3 LEGACY firebase"]`, signedIn **true**.
Branch build: `["401 tier: 1 admin", "401 tier: 2 merchant"]`, signedIn **false**, "Incorrect email or password."
Same account, same password, opposite outcome. This is the observed A/B, not an inference from the code.
Follow-up: none

4. App Check failure is refused, not downgraded [C13] ​

  • Goal: failing browser attestation must not widen what a caller can do. This is the cell that made the App Check gate on tiers 1 and 2 worth nothing.
  • Preconditions: none.
  • Steps:
    1. Launch headless chromium without injecting the App Check debug token.
    2. Attempt to sign in as the probe with the CORRECT portal password.
  • Expected: "This browser could not be verified. Reload the page and try again." No session.
  • Verify (data/console): POST /auth/admin/signin returned 401 APP_CHECK_REQUIRED, and no Identity Toolkit signInWithPassword call followed. On dev today (before this branch) the same run signs in successfully on tier 3, which is the regression this scenario pins.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    With no registered App Check debug token, the deployed build signs in anyway. The branch build stops at tier 1 with the attestation message.
Evidence:  Deployed (shipped client), unattested: `["401 tier: 1 admin", "401 tier: 2 merchant", "200 tier: 3 LEGACY firebase"]`, signedIn **true**.
Branch build, unattested, with the CORRECT portal password: `["401 tier: 1 admin"]`, signedIn **false**, "This browser could not be verified. Reload the page and try again." Tier 2 was never called.
Follow-up: This is what made the App Check gate on tiers 1 and 2 worth nothing on the shipped build: failing it routed the caller to the tier that does not attest.

5. Forced password reset cannot be sidestepped [C5] ​

  • Goal: adminPasswordResetRequired must actually block sign-in. It was a no-op: the client mapped every 428 to "no password set" and fell through.
  • Preconditions: set adminPasswordResetRequired: true on the probe (revert after).
  • Steps:
    1. Sign in as the probe with the correct portal password.
  • Expected: Your admin password needs to be reset. Use "Forgot password?" to reset it. No session.
  • Verify (data/console): POST /auth/admin/signin returned 428 ADMIN_PASSWORD_RESET_REQUIRED, no Identity Toolkit call, and adminProfiles/{uid}.failedLoginAttempts is still 0 (a correct password exiting via reset-required must not consume a strike). Restore adminPasswordResetRequired: false afterwards.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    With `adminPasswordResetRequired: true` and the CORRECT portal password, tier 1 returns 428 ADMIN_PASSWORD_RESET_REQUIRED. The deployed build swallowed it and fell all the way to legacy Firebase; the user was told "Incorrect email or passphrase.", never that a reset was required. The branch build stops at tier 1 and shows the reset instruction.
Evidence:  Deployed (shipped client): `["428 tier: 1 admin", "401 tier: 2 merchant", "400 tier: 3 LEGACY firebase"]`, error shown "Incorrect email or passphrase."
The legacy tier WAS reached. It returned 400 only because this account's legacy password differs from its portal password. An admin whose legacy password matched what they typed would have been signed in with the reset flag still set.
Branch build: `["428 tier: 1 admin"]`, error shown `Your admin password needs to be reset. Use "Forgot password?" to reset it.`
`failedLoginAttempts` was 0 before and after on both runs, confirming a correct password exiting via reset-required consumes no strike. Flag restored to `false`.
Follow-up: none

6. Lockout holds instead of falling through [C4] ​

  • Goal: the five-strike lockout only guards tier 1; falling past it to a tier with no counter is a lockout that locks nothing.
  • Preconditions: probe has a portal password. This locks a real dev account for the lockout window. Acceptable on dev; do not run against prod.
  • Steps:
    1. Sign in as the probe with a wrong password five times.
    2. Attempt a sixth sign-in, with the CORRECT portal password.
  • Expected: "Too many failed sign-in attempts. Try again later." No session, even though the password was right.
  • Verify (data/console): the sixth POST /auth/admin/signin returned 429, and no Identity Toolkit signInWithPassword call followed it.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    RUN LIVE, and the ambiguity it was meant to have turned out to be the finding. Five wrong passwords, then a sixth attempt with the CORRECT portal password. The per-IP limiter fired at attempt 3 (on the merchant path first, then the admin path), so `failedLoginAttempts` stopped at 3 and never reached five. Every 429 from that point, including the sixth attempt with the right password, was refused at tier 1 with no Identity Toolkit call. So the guard under test is CONFIRMED live for a 429, just by the limiter's 429 rather than the lockout's. The lockout-specific 429 stays covered by the emulator suite, which can reach five strikes without a shared IP in the way.
Evidence:  Live sequence, branch build, strikes read from Firestore after each attempt:

attempt 1: ["401 tier: 1 admin","401 tier: 2 merchant"] strikes=1 attempt 2: ["401 tier: 1 admin","401 tier: 2 merchant"] strikes=2 attempt 3: ["401 tier: 1 admin","429 tier: 2 merchant"] strikes=3 attempt 4: ["429 tier: 1 admin"] strikes=3 attempt 5: ["429 tier: 1 admin"] strikes=3 attempt 6, CORRECT password: ["429 tier: 1 admin"] signedIn=false legacyTierUsed=false

Strikes cleared back to 0 afterwards so the probe stays usable.
Unit: "C4: a locked account is not handed to the legacy tier" goes RED when the tier-1 guard is reverted to the shipped fallthrough. Emulator: "five wrong passwords lock the account, sequentially" and "PROBE: a BURST of wrong passwords is stopped by the five-strike gate" both pass.
Follow-up: FIXED on this branch. The client mapped ANY 429 to the lockout message, so a rate-limited admin was told they had got their password wrong five times and sent toward a reset flow they did not need. The two are distinguishable (`RATE_LIMITED` vs `ACCOUNT_LOCKED` in the response body) and are now classified separately, with a test and a mutation proof. Both stay equally terminal for the ladder, which is the part that matters.

           Incidental, not a defect: from a single IP the per-IP limiter (10 per 15 minutes, keyed per path) reaches its cap before five strikes can accumulate, so the five-strike lockout is in practice the guard for a caller spread across addresses or instances rather than for one machine guessing. Both refuse; noting it because it means the lockout path is hard to observe live and the emulator is the honest place to test it.

7. Banned admin with no portal password is refused [C7] ​

  • Goal: the ban check sits after verifyPassword, which a credential-less account never reaches, so the client got a 428 and signed the account in on legacy Firebase. A banned admin with a full session.
  • Preconditions: emulator only. Banning a real admin on dev is not a step this plan takes.
  • Steps:
    1. npm run test:admin-signin:emulator.
  • Expected: the suite passes, including "a BANNED admin with NO portal password is refused, not handed a 428".
  • Verify (data/console): the refusal is 401 INVALID_CREDENTIALS and its response body is byte-identical to the unknown-email refusal (invariant I5, a 403 here would be a ban oracle for any unauthenticated caller). The control test confirms an UNBANNED credential-less admin still gets 428, so the test is measuring the ban and not just the missing credential.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Emulator suite passes, including the new probe. Deliberately not run against a live dev admin: it would require banning one.
Evidence:  `npm run test:admin-signin:emulator`: 16 passed. New probes: banned-with-no-credential returns 401 INVALID_CREDENTIALS; the response body is byte-identical to the unknown-email refusal; the control confirms an UNBANNED credential-less admin still gets 428.
Mutation proof: setting `banned = false` in the no-credential branch turns exactly those two tests RED and leaves the other 14 green.
Follow-up: none

8. Migration window still open for a genuinely password-less admin [C6] ​

  • Goal: the regression guard. Narrowing tier 3 must not lock out the population it was written for.
  • Preconditions: an admin account with no private/auth subdoc and a working legacy Firebase password. Use a throwaway dev admin, or the probe BEFORE scenario 1 runs.
  • Steps:
    1. Sign in with the legacy password.
  • Expected: signed in normally.
  • Verify (data/console): POST /auth/admin/signin returned 428 ADMIN_PASSWORD_NOT_SET, the Identity Toolkit signInWithPassword call followed and returned 200, and POST /auth/merchant/signin was not called (a confirmed admin should not spend a rate-limited merchant call).
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    The regression guard, and the cell whose failure would lock out every admin without a portal password. The probe's `private/auth` doc was stashed to a backup file and deleted, the branch build was driven, then the doc was restored byte for byte and the restore verified with a real sign-in.
Evidence:  Branch build, no portal credential, legacy password: `["428 tier: 1 admin", "200 tier: 3 LEGACY firebase"]`, signedIn **true**.
Tier 2 was NOT called: a confirmed admin no longer spends a rate-limited merchant call. One round trip cheaper than the shipped build.
Restore verified: `["200 tier: 1 admin"]`, signedIn true, `private/auth` back with `[passwordHash, passwordSalt, updatedAt]`.
Follow-up: none

9. Merchant sign-in still works [C9] ​

  • Goal: the tier 1 to tier 2 hop on a 401 is the ladder's only legitimate fallthrough and must survive. Tier 1 answers 401 for every non-admin, so a merchant always gets one.
  • Preconditions: merchant test login from ~/.lantern-test-accounts.env.
  • Steps:
    1. Sign in to the portal as the merchant.
  • Expected: lands on that merchant's own overview.
  • Verify (data/console): POST /auth/admin/signin returned 401, POST /auth/merchant/signin returned 200, and an adminActions row with action: merchantLogin, success: true, tier: 'merchantPassword'.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Merchant signs in normally. The tier 1 to tier 2 hop on a 401 survives, which is the one fallthrough the ladder legitimately needs.
Evidence:  Branch build, merchant test account: `["401 tier: 1 admin", "200 tier: 2 merchant"]`, signedIn true.
           NOT CHECKED, same as scenario 2: the `tier: 'merchantPassword'` audit row needs the undeployed server. Worse here, because the merchant half of the server change has no emulator coverage either (`merchantSignin.integration.test.js` is untouched), so that stamp is currently unverified anywhere. Tracked as a gap, not a pass.
The page then showed "Offers are unavailable right now." That is an artifact of the local stack (the `/api/merchants` proxy points at a localhost merchants-api that was not running), not a sign-in finding.
Follow-up: none

10. Legacy tier is countable from the audit stream [I6, #924 point 2] ​

  • Goal: "how many admins are still on the legacy tier" must be answerable from data, not by opening every adminProfiles doc by hand. It was the reason #924 could not be decided.
  • Preconditions: scenario 8 has run at least once.
  • Steps:
    1. Query adminActions for action == 'adminLogin' and reason == 'ADMIN_PASSWORD_NOT_SET'.
  • Expected: one row per credential-less sign-in attempt, carrying targetUserId.
  • Verify (data/console): the distinct targetUserId set matches the accounts you know have no portal password. After this branch a 428 is the ONLY door to tier 3, so this count is exact rather than a lower bound.
Result:    [ ] pass   [ ] fail   [x] blocked
Actual:    The audit rows are a SERVER change and deployed dev is still running the previous auth-api build, so the live query cannot show them yet. Verified on the emulator instead.
Evidence:  Emulator: a credential-less sign-in writes exactly one `adminActions` row with `action: adminLogin`, `success: false`, `reason: ADMIN_PASSWORD_NOT_SET`; a successful sign-in writes one with `tier: 'adminPassword'`. Both tests were RED before the code existed.
Follow-up: Run the live query as part of scenario 11, after deploy.

11. Post-deploy: the tier discipline shipped, on a clean profile ​

  • Goal: a pre-merge pass proves the code works where it was tested. It does not prove the deployed bundle carries it. Run this against the DEPLOYED build after merge.
  • Preconditions: merged and deployed to admin.dev.ourlantern.app. Fresh browser profile, no local storage, no manual setup of any kind.
  • Steps:
    1. Load the deployed admin portal on a clean profile with the App Check debug token injected.
    2. Sign in as the probe with a WRONG password.
    3. Sign in as the probe with the CORRECT portal password.
  • Expected: step 2 refuses with "Incorrect email or password."; step 3 signs in.
  • Verify (data/console): in step 2, no Identity Toolkit signInWithPassword call. In step 3, POST /auth/admin/signin returned 200 and the adminActions row carries tier: 'adminPassword'. Record the commit and the deploy run id so a later regression can be bisected against a known-good build.
Result:    [ ] pass   [ ] fail   [x] blocked
Actual:    Runs after merge and deploy, in the post-deploy window.
Evidence:  n/a
Follow-up: none

Notes ​

Mutation proof (2026-08-20). A test that asserts an ABSENCE can pass because of a different mechanism than the one it names, so each guard was broken on purpose and the reds were recorded.

  • Client ladder. Reverting portalSignIn.js to the shipped fallthrough (tier 1 falls through on anything, tier 2 falls through to legacy on refusal) turned seven tests red. The regression guards (C1, C6 x2, C9, C10, C14) stayed GREEN, which is the right answer: they describe behaviour the shipped code also had, so a test of theirs going red would have meant the test was measuring the wrong thing. See the re-derivation note below for the current list.
  • Server ban check. Setting banned = false in the no-credential branch of adminAuth.js turned exactly two red: the banned-no-credential refusal and the byte-identical-to-unknown-email check. The other fourteen stayed green, including the control that an UNBANNED credential-less admin still gets 428, which is what proves those two are measuring the ban and not the missing credential.
  • Audit rows. Both G4 tests were red before the code existed (first TDD run: 4 failed, 12 passed), so no separate mutation was needed.

Both files were restored from byte-for-byte copies and both suites re-run green before anything was committed.

Re-derived 2026-08-20 after the review round (the counts above were recorded at 19 client tests and are restated here at the current 27): reverting the ladder to the shipped fallthrough turns 7 red, now C2+C11 (merged, see below), C4, C5, C8, C13, the rate-limited case, and the server-outage case. The regression guards stay green. Swapping adminSignIn and merchantSignIn in App.jsx turns exactly the new wiring test red; before this round nothing noticed that swap at all. Emulator: 22 passed.

C2 and C11 were merged into one test. They were byte-identical, which the review caught, and the honest reading is that at this layer an admin and a merchant with a wrong password ARE the same case: tier 1 answers the same 401 for both. Two names for one assertion inflated the red count by one and implied coverage that did not exist.

Live evidence, at a glance ​

The same account and the same credential, shipped client versus this branch:

CellDeployed build (shipped)Branch build
Legacy password, account HAS a portal password [C2]401, 401, legacy 200, signed in401, 401, refused
No App Check attestation [C13]401, 401, legacy 200, signed in401, refused with the attestation message
Reset-required set, correct portal password [C5]428, 401, legacy 400, "Incorrect email or passphrase."428, refused with the reset instruction
No portal password at all [C6]428, 401, legacy 200, signed in428, legacy 200, signed in (tier 2 skipped)
Merchant [C9]401, 200 merchant401, 200 merchant

The last two rows are the regression guards: unchanged, as intended.

Built with VitePress

Copyright © 2026 Lantern