Lantern

Appearance

AI-Prompt Onboarding Flow - Manual Test Plan ​

FieldValue
Branch / PRdocs/onboarding-fixes (post-#831 work; PR to be opened later, link this plan in its body)
EnvironmentA machine (or fresh persona shell) WITHOUT the dev toolchain, plus an AI assistant with terminal access (Claude Code, Cursor, or Copilot agent mode)
Build flags / confign/a (docs + prompt only; bootstrap-env.mjs permission-classification tweak rides along)
Build (commit)run against the branch as of 2026-08-10; doc fixes from the run land at 1c514897+
TesterMechelle (operator) ran scenarios 1 to 3 on 2026-08-10; agent filled the static check (6) and recorded evidence. Scenario 1 stays operator-run: agent runs never substitute for it.
Date2026-08-10 (first full AI-driven run)

Summary ​

#ScenarioResultNotes
1AI-driven fresh-machine setup, end to end (ACCEPTANCE, operator-run)[x] pass [ ] fail [ ] blocked2026-08-10, with 6 findings fixed mid-run
2lantern doctor passes at the finish line[x] pass [ ] fail [ ] blocked8/8 green
3Cold npm run dev with zero credentials[x] pass [ ] fail [ ] blockedHTTP 200 before credentials existed
4Wrong-shell trap: prompt refuses to install on the Windows side[ ] pass [ ] fail [x] not exercisedPositive half only (uname checks passed); trap never staged
5Re-run safety: pasting the setup twice cannot nest or damage the clone[ ] pass [ ] fail [x] partialAgent's pre-clone guard worked; double-paste untested
6Paste-safety: no inline comments in command blocks (zsh hazard)[x] pass [ ] fail [ ] blockedStatic scan of 4 files, clean (agent-run)
7Grant-pending path: bootstrap permission errors do not stall the flow[ ] pass [ ] fail [x] blockedTester is project owner; needs an ungranted account
8Activation: AI_SETUP_PROMPT.md is served in the deployed Docs tab[ ] pass [ ] fail [x] pendingRuns after the next merge + dev deploy
9Regression: the manual (non-AI) DAY_ONE path still works[ ] pass [ ] fail [x] not runLast walked 2026-08-09, before today's doc edits

Setup / preconditions ​

  • The thing under test is docs/engineering/guides/onboarding/ONBOARDING.md plus the hardened BEFORE_YOU_START.md and DAY_ONE.md.

  • Fresh-machine emulation on the dev VM (no docker/sudo needed), for agent prep runs or a quick operator dry run:

    bash
    P=~/personas/fresh-kate && mkdir -p $P/.config/gh $P/.config/gcloud $P/npm-cache
env -i HOME=$P PATH="$PATH" TERM="$TERM" GH_CONFIG_DIR=$P/.config/gh CLOUDSDK_CONFIG=$P/.config/gcloud npm_config_cache=$P/npm-cache bash --noprofile --norc

    The persona is not a separate VM or WSL instance (operator run 2026-08-10 hit this): it is the same machine and same Linux user, and it exists only inside a terminal AFTER the env -i line runs. Reopening WSL or VS Code always lands in the real home first; entering the persona is a per-terminal step. Verify you are in it: the fancy prompt disappears (--norc), echo $HOME prints .../personas/fresh-kate, and gh auth status reports not logged in.

    Caveat: PATH still carries the real user's tool installs, so "is nvm/gh/gcloud installed" answers wrong in the persona; the persona is faithful for config/auth state (git, gh, gcloud, npm cache, ~/.claude), not for binary presence. A truly bare machine (or a fresh Codespace, or a second OS user) is the honest environment for scenario 1.

  • Fresh VS Code to go with the fresh persona (operator feedback 2026-08-10 x3, see Feedback log): a plain code . reuses the already-running VS Code with the real user's extensions and AI-agent logins; the WSL code shim rejects the isolation flags; and, operator-VERIFIED, the PowerShell --user-data-dir/--extensions-dir route only isolates the Windows client side, because on connecting to WSL the extensions that actually run (Claude Code, with sessions intact) load from the shared ~/.vscode-server of the same Linux user. Working options, cleanest first:

    1. A fresh browser Codespace (github.com > Code > Codespaces > new). Nothing local is inherited, the devcontainer boots the app, and AI-agent sign-ins are genuinely first-run.

    2. A second WSL distro, the local "fresh machine" and the recommended local route:

      powershell
      wsl --list --verbose
wsl --install Ubuntu-24.04 --name kate-fresh

      (On WSL versions without --name, plain wsl --install -d Ubuntu-24.04 works when the daily distro is a different image.) First boot creates a brand-new Linux user. Expect the first shell to open at /mnt/c/Users/<you>: that is just the Windows drive as seen from Linux (WSL starts in the directory you launched from), not a sign anything crossed over; cd ~ puts you in the distro's own empty home, and nothing in the existing distro is touched. Then VS Code > New WSL Window using Distro... > the new distro. Its filesystem, home, ~/.vscode-server, and tool state are genuinely bare, so no env -i persona dance is needed there, and the AI setup prompt can install everything from zero. Delete afterward with wsl --unregister kate-fresh.

    3. Full fidelity beyond that: a second OS user or a truly bare machine. Ruled out: VS Code profiles (share account sign-ins) and the PowerShell isolated launch for WSL work (client-side only, see above); the env -i persona shell remains useful for quick SHELL-side config/auth freshness inside the main distro. Wherever the editor comes from, its integrated terminal does not inherit the persona env; paste the env -i ... line into that terminal as the first step.

  • Accounts: the tester's own GitHub login with repo access. No Google sign-in needed except scenario 7's pending-grant path. Standing guardrail: no real phone numbers anywhere near OTP tooling.

  • Clean state between runs: delete the persona dir and the test clone (rm -rf ~/personas/fresh-kate ~/repos-test/lantern_app).

Prep evidence from agent runs (recorded before the operator pass; prep only, not acceptance) ​

  • 2026-08-09, fresh-clone mock (kate-mock, this VM): npm install (2m, expected noise only), ./lantern doctor green with documented yellows, npm run dev, headless-chromium render of the landing page. Recorded as scenario 10 of ../onboarding-fixes/README.md.
  • 2026-08-09, mock desktop walkthroughs during prompt authoring surfaced the traps the prompt now encodes: wrong shell (PowerShell vs WSL), nvm installer failing to wire a missing ~/.zshrc, nested lantern_app/lantern_app clones, zsh executing pasted # comments and backticks, code not on the Mac PATH, gcloud installer's PATH prompt, VS Code/browser GitHub sessions not authenticating the terminal.
  • NOT yet exercised by an agent: the full AI-driven pass (paste prompt, assistant drives steps 1 to 11) in a bare environment. An agent may run this as additional prep in a fresh persona or Codespace; scenario 1 remains operator-run regardless.
  • 2026-08-10, FIRST LIVE AI-DRIVEN ATTEMPT (operator, kate-fresh distro, Copilot Autopilot routed to MAI-Code-1-Flash): failed at step 1. The model concatenated the whole gh install into one set -e script; the first sudo prompted for the fresh user's password; autopilot mode cancelled with "sensitive input required." Two prompt gaps fixed on the spot: a sudo hard rule (pre-authorize via user-typed sudo -v, or hand the command to the user; never take the password) and a one-step-at-a-time rule against concatenated install scripts. Re-run pending.
  • 2026-08-10, SECOND ATTEMPT (GPT-5.3-Codex via Copilot Autopilot): the model behaved well this round (uname/shell checks, guarded re-runnable keyring step, asked for sudo -v per our fix) and STILL got cancelled: sudo tickets are per-terminal, and Autopilot runs commands in its own shell, so the user's pre-auth never reaches it. Our sudo -v fix encoded a wrong assumption; prompt rewritten to make hand-the-command-to-the-user the rule after any sensitive-input cancellation. For the TEST distro only, passwordless sudo (echo "$USER ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/99-kate-testing) lets autopilot drive end-to-end; that is a test-environment convenience, never newcomer-docs material.
  • 2026-08-10, Mechelle, root-causing the cancellations herself: "it needs to stick to 'Default Permissions' if using the built in copilot agent." Confirmed: Autopilot (Preview) auto-approves but cannot pass interactive input through, so every password prompt cancels; Default permissions approves per command and lets the user type in the terminal. The orientation block now says Copilot users stay on Default permissions; the hand-the-command-over sudo rule stays as the fallback for modes without an interactive terminal.
  • 2026-08-10, Mechelle, gcloud step (curl https://sdk.cloud.google.com | bash): "Note it got stuck here. I needed to press ENTER in the terminal to actually execute it. And then I needed to press Y for the final installation. A user may not understand." Two silent hand-backs: Default permissions stages the command and waits for the user's Enter as the approval, then the installer's own Y prompts need typing. Addressed: orientation now has a "two moments are yours" bullet (stalled = click the terminal, something waits for a keypress), and a hard rule makes the assistant announce interactive installers and their expected answers before running them.
  • 2026-08-10, Mechelle, step 10 (gcloud auth login): "Did the same thing here", the staged-Enter stall repeating on a sign-in command. The interactive-command rule had named installers but only one of the three auth commands. Rule broadened to cover sign-ins explicitly (gh auth login + both gcloud auth commands), including the browser round-trip announcement and the WSL fallback (--no-launch-browser, user opens the printed URL and pastes the code).
  • 2026-08-10, RUN 2 in progress (kate-fresh2), Mechelle: "ran into the same exact need to interact with terminal issues. A user may not know that it is waiting on them. It happened on the github login as well." The announcement fix DID fire (the agent said the gcloud installer would ask in-terminal and to answer Y twice), so announcing is necessary but not sufficient: a waiting prompt and a frozen one look identical, and the user is watching the chat, not the terminal. Two changes: (a) the gcloud install is now NON-INTERACTIVE by default (bash -s -- --disable-prompts, verified against the installer's own usage text, which forwards args through the bootstrap script) with explicit PATH wiring afterward, removing the biggest stall entirely; (b) for the sign-ins that cannot avoid interactivity, the assistant must send a separate, short "ACTION NEEDED IN THE TERMINAL: ..." message immediately BEFORE running, never buried in narration, and must treat silence as waiting rather than hung. BEFORE_YOU_START's manual gcloud step matches the same non-interactive form.
  • 2026-08-10, Mechelle: "maintainer should be a separate doc vs being buried in before you start. We will likely need to adjust all the documentation tbh." First half done: the grant checklist moved to onboarding/MAINTAINER_ACCESS.md with a when-each-grant-is-needed table, BEFORE_YOU_START keeps a one-line pointer, and ENVIRONMENT_SETUP + DIRECTORY_DEFINITIONS + the AI prompt point at the new page. Second half (a broader documentation pass) deliberately NOT started mid-run; scoped as a follow-up so run 2 finishes against a stable text.
  • 2026-08-10, Mechelle, correcting the maintainer checklist: "If they have access to the admin portal they should have access to the docs.... They just need to go to admin > view docs site." Verified and she is right: admin.dev.ourlantern.app and its /docs/ path return HTTP 200 with no Access redirect, so the admin login IS the docs grant and my "add them to a Cloudflare policy" step was wrong. The separate Access-gated docs.dev.ourlantern.app host is a different, optional route (for handing someone docs WITHOUT an admin account). MAINTAINER_ACCESS rewritten around that, and the pre-repo pointers in BEFORE_YOU_START and DAY_ONE now name the admin portal path. Her follow-on ask (a terminal one-liner for Cloudflare Access) is documented in a collapsed section with the honest shape: the Access API REPLACES a policy's include list, so it is a read-append-write, not a one-liner; the repo token is confirmed read-capable for Access apps and policies, write scope unexercised, and no policy was mutated from here because it is an authorization surface.
  • 2026-08-10, Mechelle, after watching a run finish: "I think we need more frenly output after the run finalizes. Such as how to initialize and use the lantern app, maybe have an onboarding skill to establish the user specific repo for lantern-context (voice, preferences, etc), maybe next steps on PR instructions or how to push a feature." Addressed: the prompt gains a step 12 hand-off (what is running and where, try the core venue/lantern loop, signup is invite-gated so ask for a link, run /context-onboard for personal AI context, the first-PR shape in one breath, what is still pending on a maintainer, then ask what they want to see first), and DAY_ONE gains a matching "You are in. What now?" section for the hand-run path. Note /context-onboard already exists and is machine-local; the separate lantern_context REPO is another session's in-flight work (#834) and is deliberately not wired into onboarding text yet.
  • 2026-08-10, Mechelle: "Why not also the reload window and confirm the lantern extension actually installed?" Right, and run 1's transcript shows the cost: the agent's final summary listed "manual UI confirmations still needed (reload, confirm the icon)" and handed that to the user as homework. Step 9 now states that a CLI listing is not proof (VS Code hides a freshly installed extension until a window reload), tells the assistant to walk the user through the reload and get a yes/no on the icon, and to re-run the --force install rather than close with an open item. The hand-off also points them into the sidebar and what it does.
  • 2026-08-10, Mechelle: "The lantern extension has a 'Start All' that actually runs and opens the dev server across EVERYTHING. Whereas the introduction only opens the app..." Real mismatch: onboarding taught npm run dev (app only) and then handed people a sidebar whose Quickstart starts app + admin + all APIs + docs site + public site + Storybook + a test watcher. Both the doc and the AI prompt now name the difference, say to stay on npm run dev until the rest is actually needed, and carry the reason emulators are excluded from Quickstart (Firestore emulator and venues API both bind 8080).
  • 2026-08-10, Mechelle, reading the merged page: "This doesn't make sense. The user would have to get admin access in order to access this onboarding doc...." Circular, and she is right twice over: the line addressed a reader who by definition is already reading the page, and its fallback (admin portal docs) needs a grant that is MORE work to issue than the GitHub invite. Removed the line; the distribution answer now sits with the person who actually acts on it, in the POC section: do the GitHub invite first (seconds, and then they read it on github.com), with the self-contained AI prompt block as the paste-into-a-message option. The docs-site route stays documented as a later convenience for machines without a clone, explicitly not as the pre-repo path. Also caught in the same pass: the merge had left two near-identical "Maintainers:" notes and a stale "Part 1" cross-reference.
  • 2026-08-10, Mechelle: "How would user know where to put the prompt? We might need to share screenshots or direct them where stuff is in VSCode (the agent and the terminal actually)." Addressed with a "Where things live in VS Code" orientation block in AI_SETUP_PROMPT.md (opening the chat panel, picking Agent mode, the terminal, the approve-and-done rhythm). Screenshots deliberately deferred: UI chrome churns fast and rots screenshots; revisit if text directions prove insufficient in the next run.

Scenarios ​

1. AI-driven fresh-machine setup, end to end (ACCEPTANCE, OPERATOR-RUN) ​

  • Goal: a real person with a machine lacking the toolchain reaches a running app by pasting the prompt and approving as the assistant works.
  • Preconditions: machine (or bare VM/Codespace/second OS user) without gh, nvm, Node, gcloud, or a clone; VS Code + one terminal-capable AI assistant installed; GitHub account with repo access.
  • Steps:
    1. Open AI_SETUP_PROMPT.md (via the Docs tab if no repo access) and copy the entire fenced block.
    2. Paste it into the assistant and follow along, approving each install as it proposes it.
    3. Let it run steps 1 through 11 without hand-holding; intervene only when it asks.
    4. Note every place it stalls, guesses wrong, or needs an unprompted correction.
  • Expected: assistant checks-before-installing, narrates one line per step, lands the finish line (doctor + npm run dev); it pauses rather than improvises when something fails twice.
  • Verify (data/console): ~/repos/lantern_app exists with no nested clone; gh auth status succeeds; node --version matches .nvmrc; Lantern Control icon in the Activity Bar.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    2026-08-10, OPERATOR RUN on the kate-fresh WSL distro (new Linux user "meche"),
           Copilot agent, GPT-5.3-Codex, after switching Autopilot -> Default permissions.
           Reached the finish line: clone to ~/repos/lantern_app, nvm install from .nvmrc
           (Node 24.16.0, npm 11.13.0), npm install, npm link (bare `lantern` resolved),
           gcloud 579.0.0, Lantern Control installed, doctor fully green, dev server up,
           curl http://localhost:5173/ -> HTTP 200. Roughly 12 minutes of agent work
           across 4 turns, with the operator answering prompts.
           Six findings, each fixed on the branch as it surfaced (details in Feedback log):
           1. Autopilot mode cancels on any password/interactive prompt -> use Default
              permissions (operator root-caused).
           2. Concatenated `set -e` install scripts die whole on one prompt -> one step
              at a time.
           3. sudo pre-auth via `sudo -v` does NOT reach the agent (per-terminal tickets)
              -> hand sudo commands to the user.
           4. Staged commands wait for the user's Enter, and installers ask their own Y/N
              -> assistant must announce interactive commands.
           5. `code --install-extension ms-vscode-remote.remote-wsl` FAILS from inside WSL
              ("declared to not run in this setup"). This was a WRONG INSTRUCTION added to
              our own docs earlier the same day; corrected in BEFORE_YOU_START, DAY_ONE,
              and the prompt (it is a Windows-side extension; already present whenever a
              WSL window opens with the badge).
           6. An interrupted gcloud installer leaves the binary on disk but no PATH wiring;
              the agent self-healed by appending path.bash.inc, now an explicit prompt rule
              so weaker models do the same instead of re-running the installer.
           Also observed, no action needed: ADC login failed once with `missing_code` after
           the browser hand-off and succeeded via --no-launch-browser (the fallback added
           to the prompt earlier that day, validated in the wild); every agent command runs
           in a fresh shell, so it correctly prefixed `source ~/.bashrc`; a bare
           `lantern-control` CLI check misses the publisher-qualified id
           `cattreedev.lantern-control` (now documented).
Evidence:   run-1-2026-08-10-transcript.md in this folder (distilled tool-call timeline,
           4 turns / 65 log entries). The raw 11.7 MB Copilot export is gitignored and
           kept locally; the distilled file carries every command, intent, and response
           head.
Follow-up:  Re-run once on a NEW disposable distro against the corrected prompt to confirm
           a clean pass with no mid-run fixes; that is the regression baseline.

2. lantern doctor passes at the finish line ​

  • Goal: the health check agrees the setup is complete.
  • Preconditions: scenario 1 finished (or the manual path of scenario 9).
  • Steps:
    1. Run lantern doctor (bare; ./lantern doctor from the repo root if the link was skipped).
  • Expected: exit 0, "Environment looks good." If the secret grant is still pending, ONLY the .env.local and gcloud-auth rows are yellow; lantern on PATH is green and points at THIS clone.
  • Verify (data/console): the doctor output block, saved into Evidence.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Final run: 8/8 green ("Environment looks good"), including `lantern on PATH
           linked, use `lantern <cmd>` anywhere` from the bare command in the new distro.
           The documented PENDING state was also observed earlier in the same run (before
           ADC login and env:bootstrap): exactly two yellows, `gcloud auth (ADC) not
           authenticated` and `.env.local missing`, each printing its next command, with
           the overall summary still "Environment looks good." Both halves of scenario 3's
           promise (yellow rows do not block) therefore verified in one run.
Evidence:   run-1 transcript, 19:43:01 (two yellows) and 19:47:03 (all green).
Follow-up:  none

3. Cold npm run dev with zero credentials ​

  • Goal: the app runs before any Google sign-in or secret grant exists.
  • Preconditions: clone + install done; NO .env.local, no gcloud auth performed.
  • Steps:
    1. npm run dev from the repo root.
    2. Open http://localhost:5173.
  • Expected: landing page renders (hero, Browse Places, Create Account). No credential prompt, no blocking error.
  • Verify (data/console): browser console shows Firebase initialized against lantern-app-dev.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Dev server started 19:43:14 and `curl -I http://localhost:5173/` returned
           HTTP/1.1 200 OK at 19:43:19. Credential-free is proven by ordering, not by
           assertion: at 19:43:01 doctor showed `.env.local missing` and ADC not
           authenticated, and the first `gcloud auth login` did not run until 19:44:15.
           So the app served a page BEFORE any Google sign-in or secret existed on the
           machine.
Evidence:   run-1 transcript, timestamps 19:43:01 / 19:43:14 / 19:43:19 / 19:44:15.
Follow-up:  A browser render (not just HTTP 200) is still unproven on this distro; the
           2026-08-09 kate-mock run covered the visual render on the main machine.

4. Wrong-shell trap: prompt refuses to install on the Windows side ​

  • Goal: on Windows, the assistant detects a non-WSL shell and routes into WSL before installing anything.
  • Preconditions: Windows machine (or skip as blocked on Mac-only availability).
  • Steps:
    1. Paste the prompt into an assistant whose terminal is PowerShell.
    2. Watch its first platform check.
  • Expected: it runs uname, notices Windows, and helps enter/install WSL; it does NOT apt/choco/winget-install any project tooling on the Windows side.
  • Verify (data/console): first tool installs happen only after uname prints Linux.
Result:    [ ] pass   [ ] fail   [x] NOT EXERCISED (no negative case available)
Actual:    Only the positive half ran: the agent checked `uname -s` -> Linux and `uname -r`
           -> microsoft-standard-WSL2, concluded correctly that it was inside WSL, and
           detected bash -> ~/.bashrc before installing anything. The trap itself (starting
           the agent in a PowerShell terminal and watching it refuse to install Windows-side)
           was never staged, because the whole run happened inside the WSL distro.
Evidence:   run-1 transcript, environment-check step of the second attempt.
Follow-up:  Stage it deliberately on the next run: open the agent with a PowerShell terminal
           active and paste the prompt.

5. Re-run safety: pasting the setup twice cannot nest or damage the clone ​

  • Goal: the guarded clone block and re-runnable steps survive a second pass.
  • Preconditions: a completed setup from scenario 1 or 9.
  • Steps:
    1. From ~/repos, paste DAY_ONE's clone-and-install block again in full.
    2. Then cd ~/repos/lantern_app and paste it a third time from INSIDE the clone.
  • Expected: no lantern_app/lantern_app appears; the clone line skips itself both times; install/link/code steps re-run harmlessly ("already exists" answers count as done).
  • Verify (data/console): find ~/repos/lantern_app -maxdepth 1 -name lantern_app returns nothing.
Result:    [ ] pass   [ ] fail   [x] PARTIAL (guard logic exercised, double-paste not)
Actual:    The prompt's anti-nesting intent worked in practice: before cloning, the agent ran
           `find "$HOME" -type d -name lantern_app` (no output) and `pwd` (/home/meche), said
           in so many words that no existing clone was found and the cwd was safe, then cloned
           into ~/repos/lantern_app. What was NOT tested is the mechanical guard in DAY_ONE's
           block: pasting the clone-and-install block a second time from ~/repos, and a third
           time from inside the clone.
Evidence:   run-1 transcript, 19:31:53 (find + pwd) and 19:32:04 (clone).
Follow-up:  Run the two extra pastes on the existing kate-fresh clone; it is non-destructive
           by design, so it can be done any time that distro is alive.

6. Paste-safety: no inline comments in command blocks (zsh hazard) ​

  • Goal: no command block in the onboarding docs can misfire when pasted into macOS zsh (where # is not an interactive comment and backticks execute).
  • Preconditions: none (static check; agent-runnable).
  • Steps:
    1. `grep -n '#' inside every fenced ```bash block of BEFORE_YOU_START.md, DAY_ONE.md, AI_SETUP_PROMPT.md, and README.md's Start-here block; ignore shebangs and heredoc content that is deliberate.
  • Expected: zero inline # comments and zero backticks inside pasteable command lines; explanations live in prose (the EDITORS comment in DAY_ONE states the rule).
  • Verify (data/console): grep output pasted into Evidence.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Static check run by the agent 2026-08-10 against BEFORE_YOU_START.md, DAY_ONE.md,
           AI_SETUP_PROMPT.md, and README.md: zero inline `#` comments and zero backticks
           inside pasteable command blocks. Note this is a repo-side check, not a macOS
           behavioral test: it proves the hazard is absent from our copy, not that zsh
           behaves as described (that part is documented, not re-verified here).
Evidence:   fenced-block scanner over the four files, all clean; EDITORS comment in DAY_ONE
           states the rule for future editors.
Follow-up:  Worth wiring into the docs linter eventually so a future edit cannot reintroduce
           it silently.

7. Grant-pending path: bootstrap permission errors do not stall the flow ​

  • Goal: with no secretmanager.secretAccessor grant, the assistant reports the pending grant and continues; nothing claims the app is blocked.
  • Preconditions: an account WITHOUT the grant (or a persona with fresh ADC); reaches step 10 of the prompt.
  • Steps:
    1. Complete gcloud auth login + gcloud auth application-default login as the ungranted account.
    2. Run npm run env:bootstrap.
  • Expected: every secret reports a permission error categorized as a pending grant (not "unknown"); the assistant relays "request the grant, continue meanwhile" and proceeds to doctor + dev; the app still runs (scenario 3 behavior).
  • Verify (data/console): bootstrap summary line counts the failures as permission, not unknown (this run also exercises the permission_denied classification fix in bootstrap-env.mjs).
Result:    [ ] pass   [ ] fail   [x] BLOCKED (no ungranted account available)
Actual:    Not testable on this run: the tester is the project owner, so `env:bootstrap`
           succeeded outright and doctor ended with `.env.local present, in sync with
           example`. The adjacent half DID get verified: before bootstrap, doctor showed the
           documented pending-state yellows and the run continued to a working app, which is
           the behavior this scenario ultimately protects.
Evidence:   run-1 transcript, doctor at 19:43:01 vs 19:47:03.
Follow-up:  Needs the actual newcomer (or a throwaway Google account with no IAM binding) to
           exercise the permission-error path end to end, including the bootstrap summary
           classifying failures as `permission` rather than `unknown`.

8. Activation: AI_SETUP_PROMPT.md is served in the deployed Docs tab ​

  • Goal: the no-repo-access reader can actually reach the prompt (capability vs activation: the file existing in the repo proves nothing about the served site).
  • Preconditions: the branch merged and the dev deploy landed.
  • Steps:
    1. Sign in to the dev admin portal (needs the Cloudflare Access grant), open Docs > Guides > Onboarding.
    2. Open AI Setup Prompt; copy the block from the rendered page.
  • Expected: the page renders with the full fenced block intact and copyable; BEFORE_YOU_START's section 7 links to it.
  • Verify (data/console): rendered URL noted in Evidence with the deploy run id.
Result:    [ ] pass   [ ] fail   [x] PENDING (branch not merged yet)
Actual:    Cannot run: the deployed Docs tab serves origin/dev, and this work sits on
           docs/onboarding-fixes with its PR still to be opened. The pre-merge half is
           verified: the file exists at docs/engineering/guides/onboarding/ONBOARDING.md
           and BEFORE_YOU_START section 7 links to it, and the equivalent activation check
           passed for the three onboarding pages after PR #831 merged.
Evidence:   local docs build after the #831 merge rendered guides/onboarding/ correctly.
Follow-up:  Run after the next merge + dev deploy; also confirm the fenced block survives
           VitePress rendering intact and is copyable.

9. Regression: the manual (non-AI) DAY_ONE path still works ​

  • Goal: the hand-driven path the docs teach is not broken by the AI-flow edits.
  • Preconditions: fresh clone location (persona shell is fine).
  • Steps:
    1. Follow DAY_ONE literally, top to bottom, without an assistant: guarded clone block, install, link, code ., doctor, dev, extension install.
  • Expected: identical outcome to before the AI-flow work: running app, green doctor (expected yellows only), Lantern Control installed.
  • Verify (data/console): same checks as scenarios 2 and 3.
Result:    [ ] pass   [ ] fail   [x] NOT RUN THIS CYCLE
Actual:    The manual path was last exercised 2026-08-09 on the kate-mock clone (install,
           doctor, dev server, browser render) BEFORE today's prompt-driven edits to
           DAY_ONE and BEFORE_YOU_START. Today's run drove the AI path only, so the
           hand-driven sequence has not been walked against the current text.
Evidence:   scenario 10 of ../onboarding-fixes/README.md for the 2026-08-09 manual pass.
Follow-up:  Walk it once on a fresh distro without an assistant; this is the cheaper of the
           two remaining runs and catches copy that only makes sense to an AI reader.

Notes ​

Operator-run boundary (standing preference, 2026-08-09): agent mock runs prepare for and de-risk this plan; the acceptance pass in scenario 1 is performed by Mechelle or the new colleague, and an agent must never mark it passed on their behalf.

Run 2 (second fresh instance, PENDING) ​

One plan, one folder: re-runs are recorded here as dated run sections rather than a v2 file, so the scenario definitions never fork and the history of what each run proved stays in one place. Run 1's results above stay verbatim as the baseline; record run 2's outcomes in this section, and only edit a scenario block above if the scenario's DEFINITION changes.

Purpose: confirm a clean pass against the corrected prompt (run 1 needed six mid-run fixes, so it proves the path works, not that the shipped text is sufficient), and close the scenarios run 1 could not.

Setup: a brand-new distro, so the corrected docs are exercised from zero:

powershell
wsl --unregister kate-fresh
wsl --install Ubuntu-24.04 --name kate-fresh2

Copy the CURRENT AI_SETUP_PROMPT.md block (it changed six times on 2026-08-10), open a VS Code window on the new distro, Copilot on Default permissions, and paste.

Must cover:

  1. Scenarios 1 to 3 again, expecting NO mid-run doc fixes. Any fix needed is the finding.
  2. Scenario 4 (wrong-shell trap): start the agent with a PowerShell terminal active before pasting, and watch it route into WSL rather than installing Windows-side.
  3. Scenario 5 (re-run safety): paste DAY_ONE's clone-and-install block a second time from ~/repos, then a third time from inside the clone; confirm no nested lantern_app/lantern_app.
  4. Scenario 9 (manual regression): optionally walk DAY_ONE by hand instead of via the agent on this same distro, to test the human-readable path against the current text.
  5. Watch specifically whether the WSL-extension correction reads right from inside a WSL window, since that instruction was wrong in run 1.

Still not coverable locally: scenario 7 (needs an account without the Secret Manager grant, i.e. the actual newcomer) and scenario 8 (needs this branch merged and deployed).

Result:    [ ] pass   [ ] fail   [ ] blocked
Date:
Actual:
Evidence:   (export a fresh Copilot transcript and distill it beside this file)
Follow-up:

Feedback log (operator feedback lands here as it arrives; paper-trail rule set 2026-08-10) ​

  • 2026-08-10, Mechelle, on the persona recipe: copied the fresh-persona shell block and asked "wouldn't we also want to open this new user in VS Code so they can access the AI agent directly?" Correct: a plain code . would reuse the running VS Code with real-user state, defeating the persona. Addressed by adding the isolated-VS-Code recipe (--user-data-dir/--extensions-dir + -n) to Setup, with the note that its integrated terminal needs the persona env pasted in.
  • 2026-08-10, Mechelle, ran that recipe: the WSL code shim rejects both flags ("Ignoring option 'user-data-dir': not supported for code"). Recipe was wrong for this machine; replaced with the ranked options above (Codespace / PowerShell-side launch / second OS user) and the note that VS Code profiles do not isolate sign-ins.
  • 2026-08-10, Mechelle: "How would the user see ai_setup_prompt.md if they don't have access to it? Or bare minimum, the lantern admin?" Real distribution gap: pre-repo the only route was the Docs tab, which itself needs two grants. Addressed: the prompt file and BEFORE_YOU_START now say a maintainer can simply copy the self-contained block into an email/DM; nothing in it requires repo access to read.
  • 2026-08-10, Mechelle: "Just realized that code . opens to windows, in our case we want it to open in WSL." Root cause: without the ms-vscode-remote.remote-wsl extension, code . from WSL opens a Windows-local window over the network share. Addressed: BEFORE_YOU_START installs the extension by command line with the green "WSL: Ubuntu" badge as the tell, DAY_ONE's code . prose carries the badge check + one-line fix, and the AI prompt's step 8 makes the assistant verify the badge on WSL.
  • 2026-08-10, Mechelle, clarifying the code . finding: the opens-to-Windows report was about the LOCAL VM TESTING flow, not (only) the newcomer docs. The docs additions stand (she confirmed they are good to have), and the Setup recipe's Windows-side launch now carries the missing connect-to-WSL steps (install the WSL extension in the isolated window, palette "WSL: Connect to WSL", badge check, avoid code . while two instances exist). She also flagged that BEFORE_YOU_START needs better mac-vs-windows separation; shape question put to her, she chose per-step OS labels, and the page now labels every OS-specific step Mac / Windows (WSL) / Both through sections 2 to 5.
  • 2026-08-10, Mechelle, mid-run: reopened WSL in VS Code and "I think I am back in my main WSL folder structure... not the VM we created." Expected behavior, but the recipe never said so: the persona is not a VM, it is a per-terminal env, and every new terminal starts in the real home until the env -i line runs. Setup now states this plus a verify-you-are-in-the-persona check (plain prompt, echo $HOME, gh auth status logged out). Her screenshot also showed the MAIN VS Code (Claude Code panel present), not the isolated instance; noted that the shell-side flow can be tested from the main editor, editor-side freshness cannot.
  • 2026-08-10, Mechelle: "But I did run the powershell code . open then connect to WSL..." She had followed the isolated-launch route correctly, and her window still showed Claude Code with her sessions: operator-verified proof that the client-side flags do not isolate WSL-remote work (extensions run from the shared ~/.vscode-server). Recipe re-ranked: a second WSL distro (wsl --install Ubuntu-24.04 --name kate-fresh, then "New WSL Window using Distro...") is now the recommended local fresh-machine, matching her original "the VM we created" intuition; the PowerShell flags route is ruled out for WSL work.
  • 2026-08-10, Mechelle, first kate-fresh boot: "did I mess up my existing users mnt?" No: the new distro's first shell opened at /mnt/c/Users/meche because WSL starts in the launching directory, and /mnt/c is the shared Windows drive, not the existing distro. Nothing was modified. Recipe now warns the first shell lands there and says cd ~ first. (She created the new user as "meche" rather than "kate"; harmless, still a bare user.)
  • 2026-08-10, Mechelle: "there should be a line in the instructions somewhere or provided by the AI prompt that an existing dev admin would run in the terminal to provide login access to everything (GCP, Github, Prelude, etc)". Addressed: BEFORE_YOU_START gained a "For the maintainer" grant checklist (gh collaborator invite + gcloud IAM grant as runnable commands; Cloudflare Access, admin-dev login, app invite, and Prelude as dashboard steps, which have no CLI), ENVIRONMENT_SETUP's grant section points at it, and the AI prompt's step 10 tells the assistant where the maintainer commands live.

Built with VitePress

Copyright © 2026 Lantern