Lantern

Appearance

2026-04-29 — Issue #337 Cloud Run Deploy Gaps: Shipped + Lessons ​

Outcome: All four #337 acceptance items shipped via PRs #338, #339, #345, plus an outstanding GitHub-API rewrite (this worklog's branch). Five follow-up issues filed (#340, #342, #343 + the IAM-binding work + the SW staleness fix). Today exposed a number of process gaps worth writing down so I don't repeat them.

What shipped ​

PRBranchWhat it did
#338chore/preview-label-gateGate deploy-preview.yml on a preview PR label so 4 Cloudflare Pages previews don't fire on every PR push. Also catalogs the new label in triage-categories.json excludedLabels.
#339fix/337-cloud-run-deploy-gapsThe headline #337 work: registry devUrl for auth-api + lanterns-api + assistant-api; merchants-api Dockerfile + deploy job; assistant-api deploy job (initial, snapshot-bundling design); <LanternChat /> UX guard for unset VITE_ASSISTANT_API_URL; admin-deploy-job env wiring for assistant URL.
#341fix/337-followup-assistant-deployFirst follow-up — fixed cp -r services recursing into the snapshot destination via mktemp -d + mv. Set merchants-api devUrl (deferred in the original PR for no good reason).
#345fix/337-revert-iam-stepRevert of an IAM-binding step I added across all 7 deploy jobs in PR #344. gcloud add-iam-policy-binding failed with PERMISSION_DENIED in CI because the WIF service account lacks run.services.setIamPolicy.
(this branch)fix/337-assistant-github-runtimeRip out the repo-snapshot bundling design entirely. New repoFs.js reads via the GitHub Contents/Search API at runtime. Smaller image, no per-deploy snapshot churn, content stays fresh between deploys. Includes a test-repofs.mjs smoke-test script.

What broke + why ​

The #337 work surfaced five distinct failures before everything was healthy. Three of them were on the assistant-api deploy alone:

  1. cp recursion in the snapshot stage — destination services/api/assistant/repo-snapshot/ lives inside services/, so cp -r services … recursed into itself. Fixed in #341.
  2. cp failed on missing allow-list paths — I copied the repoFs.js allow-list verbatim into the workflow, but .github/instructions/ and AGENTS.md don't exist in the tree. Fixed (defensively) by looping with [ -e ] checks.
  3. gcloud --source excluded the staged dirs from the build context — the per-service .gitignore lists .shared-pkg/ and repo-snapshot/, and gcloud run deploy --source falls back to .gitignore when no .gcloudignore exists. Build context was 115KB instead of expected ~MB; Docker COPY .shared-pkg/ failed. Fixed by adding an explicit .gcloudignore (and ultimately by removing the snapshot pattern entirely in this branch).
  4. IAM binding step PERMISSION_DENIED — I added a gcloud run services add-iam-policy-binding step to all 7 deploy jobs to work around --allow-unauthenticated being silently no-op'd on this account. It worked locally with my admin creds; failed in CI because the WIF service account only has deploy perms, not setIamPolicy. The actual gcloud run deploy succeeded, so services kept serving — but my added step exited non-zero, making 5 previously-green deploy jobs report failure. This is the one that genuinely scared us. Reverted in PR #345.
  5. Localhost-fallback CSP error in merchantsApi.js — admin client fell back to http://localhost:8085 in prod when VITE_MERCHANTS_API_URL was unset; CSP correctly blocked. Same shape of bug latent in assistantApi.js (empty-string fallback → relative URL → SPA → 405). Both fixed.

Lessons ​

These are the things I want to internalize so I don't burn another deploy cycle on them.

1. Local creds ≠ CI creds. Test the actual CI auth path. ​

The IAM step (#4 above) is the hardest one to swallow. I tested gcloud add-iam-policy-binding locally, it worked, I shipped it. It worked because my gcloud was authenticated as me — an admin. CI runs as a Workload Identity Federation service account with a much narrower role set.

Going forward: when adding any new gcloud step, look up which role grants the operation (here: roles/run.admin for setIamPolicy) and confirm the WIF SA has it before merging. If you can't read the WIF SA's IAM directly, at minimum document the assumption in the commit message so the failure is traceable.

2. Dry-run the actual bash on a clean tree before pushing. ​

Bugs #1 and #2 (the cp failures) were both shell-level. They would have been caught in seconds by:

bash
# in a clean checkout, paste the run: block exactly as it appears in the workflow
bash -c '<the full script>'

I did this once after #341 and it would have caught the .github/instructions failure before the push. I just didn't do it consistently. From now on: any new shell in a workflow gets dry-run locally on a clean checkout before commit. No exceptions.

3. gcloud run deploy --source upload uses .gitignore by default. ​

Worth knowing as a first-class fact. If you stage runtime artifacts into a build context that are gitignored (and you don't have a .gcloudignore), gcloud silently drops them. Symptom: tiny build context + COPY failure inside Docker. Always pair runtime-staged artifacts with an explicit .gcloudignore that overrides the relevant .gitignore exclusions.

4. When a substantial change ships, ship a way to test it locally. ​

The test-repofs.mjs script in this branch is the model for what I should have done with the snapshot logic too. A 100-line smoke test that you can re-run after every change catches a class of bugs that no amount of YAML linting or local script-testing will catch.

Pattern: for any change that touches a backend service's behavior, ship a test-<thing>.mjs (or equivalent) alongside the change that exercises the new code path against a realistic environment. Calling it out in the PR description makes the reviewer's job 10× easier.

5. Bundle related work into a single PR. ​

I split the original #337 work into two PRs (the gate detour + the main work) when one would have done. That created the very mess the user objected to: orphan PRs, ordering issues, and a fragile "land #338 first to save preview spend" plan that didn't hold up. Saved as a feedback memory.

6. Don't trust apparent damage without checking. ​

When the IAM step failed, the GitHub Actions UI showed 5 red X's and the user understandably panicked. The actual gcloud run deploy had succeeded for all five — only my added IAM step had failed. All services were still serving 200. I should have led with curl /health checks rather than reverting blind.

This is also a CI design concern: a post-deploy validation step that fails should not make a successful deploy report failure. We should split "deploy" and "validate" into separate GitHub Actions jobs (or at least separate steps with continue-on-error: true on the validation) so the apparent-vs-actual damage signals stay aligned.

7. Localhost fallbacks are a liability. ​

Two API clients had silent localhost / empty-string fallbacks that bit users in prod. The pattern across the codebase is inconsistent: 3 clients fall back to a Cloud Run dev URL, 2 fall back to localhost-or-empty, 4 require an env var. Issue #343 is the proper fix — registry-driven getApiBaseUrl(slug, envVar) helper that fails loudly in misconfigured prod builds.

8. Worktrees aren't always worth it for sequential PR work. ​

We started in a worktree to keep the main checkout clean while we worked on #337. Hit two real frictions: .env.local doesn't transfer, and Claude Code's per-project context (memory, allowed permissions, hooks) splits across worktree paths. Switched back to plain branches in the main checkout, which was strictly better for this workflow. Saved as a feedback memory.

Open follow-ups (issues filed) ​

  • #340 — validate orchestrator regenerates version.json + client SDK on every run. Stop mutating in validate; only check freshness.
  • #342 — Admin PWA serves stale JS bundle until tabs fully closed. Add skipWaiting + clients.claim() or a "new version available" banner.
  • #343 — Refactor 9 API client base-URL definitions to a single registry-driven helper.
  • (unfiled) — IAM binding for new Cloud Run services should be a one-time setup task per service, not a per-deploy step. Either grant the WIF SA roles/run.admin, or document a manual gcloud run services add-iam-policy-binding runbook step for each new service.
  • (unfiled) — octokit /search/code endpoint deprecation 2026-09-27. The new repoFs.js will need to switch to the GraphQL search API or the announced replacement before then.

Files in this branch ​

Built with VitePress

Copyright © 2026 Lantern