Lantern

Appearance

Agent Skills + Instructions Refactor - Manual Test Plan ​

FieldValue
Branch / PRclaude/repo-memory-skills-org-p0owiz (draft PR pending)
Environmentrepo root for the check commands; a fresh Claude Code session (and optionally Copilot / Gemini) for the behavior checks. No app/services needed.
Build flags / confign/a
Build (commit)c6a1c371
TesterMechelle
Date2026-07-13

Scope: the agent-facing instruction layer this branch reworks, NOT app behavior. That is:

  • 10 skills changed under .agents/skills/ (source of truth): 6 new (cloud-service, code-comments, curate-memory, debug, privacy-architecture, env-vars), design and pr-workbench expanded, monitor-ci rewritten for GitHub Actions, test-plan tweaked.
  • AGENTS.md as canonical, imported into CLAUDE.md via a bare @AGENTS.md line; CLAUDE.md and .github/copilot-instructions.md reduced to thin pointers.
  • Distribution: npm run sync:skills fans each skill out to .claude/skills/, .github/skills/, .gemini/skills/, plus .github/prompts/ + .gemini/commands/ slash-shims for command: true skills.
  • Operator Voice output style and the docs/fireside/ comms/origins content.

Update (2026-07-13): the env-vars skill was authored on the C3 branch and was briefly dropped when the C3 stack was rebased off; it has since been restored to this branch (it is a general agent skill), so it is now present and part of the distribution (scenario 11). Two app-code "creep" commits (App Check, admin metrics) were also dropped, so the branch is now purely the agents restructure.

Summary ​

#ScenarioResultNotes
1npm run validate green (guards the instruction layer)[x] passfull suite 29/29 green (after fixing a rebase-artifact em-dash baseline)
2Skill Sync: source and all harness copies in lockstep[x] passsource and all harness copies in lockstep; now committed
3Distribution: 4 harnesses + slash shims + AUTO-SYNCED banner[x] passverified via env-vars restore (all 3 skill dirs + both shims + banner)
4@AGENTS.md import loads canonical rules in a fresh session[x] passfresh session quoted rules 16 + 9 verbatim and correctly
5Pointer files consistent (CLAUDE / Copilot / Gemini -> AGENTS.md)[x] passGEMINI.md now created; all three import/point to AGENTS.md, no dup rules
6New skills auto-fire on their triggers and carry guardrails[x] pass5/6 fired cold (cloud-service, privacy w/ pushback, curate-memory, debug, env-vars); code-comments deferred to a target
7Expanded skills carry the new content (design, pr-workbench)[x] pass/design + /pr-workbench loaded the expanded content (Storybook/no-purple/group-by-feature; origin/dev/draft/worktree)
8monitor-ci rewrite targets GH Actions; NX + poll scripts gone[x] passpoll scripts gone, no nx/poll refs in the skill
9Guardrail/negative: a skill makes the agent REFUSE / redirect[x] passpassphrase refused (privacy axiom); Cloud Function redirected (rule 9)
10Operator Voice output style loads and shapes replies[x] passrecommendation-first + decision matrix + zero em dashes in the reply
11Regression: unchanged skills load; env-vars restored + present[x] passenv-vars restored across all harnesses; untouched skills unaffected

Setup / preconditions ​

  • Run check commands from repo root on branch claude/repo-memory-skills-org-p0owiz at c6a1c371.
  • For behavior scenarios (4, 6, 7, 9, 10), start a fresh Claude Code session in this repo so the reworked instructions load clean. For cross-harness spot checks (3, 5), Copilot and Gemini are optional but stronger.
  • "Auto-fire" means the agent invokes a skill on its own from the task description (the skill's USE WHEN / description), not only when slash-invoked. Test both: describe a matching task and watch, then also /skill-name it.
  • No services, no Firebase, no destructive steps in this plan. Everything is read-only against the repo and agent behavior.
  • Skills changed on this branch: cloud-service, code-comments, curate-memory, debug, privacy-architecture, env-vars (new); design, pr-workbench (expanded); monitor-ci (rewritten); test-plan (tweaked).

Scenarios ​

1. npm run validate green (guards the instruction layer) ​

  • Goal: The gate that protects the instruction layer (Skill Sync, em-dash ratchet, secret-scan, lint) is fully green on this branch.
  • Preconditions: clean tree at c6a1c371.
  • Steps:
    1. From repo root, run npm run validate.
    2. Read the summary table.
  • Expected: all sections pass, exit code 0. In particular Skill Sync, Em Dash Check, and Secret Scan pass.
  • Verify (data/console): summary shows no failing row; echo $? is 0.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    First run failed only on Em Dash Check, a rebase artifact (not a code
           defect): the kept baseline commit had tightened Chat.jsx's allowance
           to 2 from the dropped C3 version, while the file reverted to dev's 3.
           Baseline restored to 3. After that a FULL `npm run validate` ran
           29 passed, 0 failed (118s), and it stayed green after dropping the two
           app-code creep commits and restoring env-vars + GEMINI.md.
Evidence:  Final summary "Total: 29 passed, 0 failed" / "VALIDATION PASSED".
Follow-up: none.

2. Skill Sync: source and all harness copies in lockstep ​

  • Goal: Every .agents/skills/<name>/skill.md source matches its generated copies; no drift after the rebase (which forced us to hand-resolve the design-skill files).
  • Preconditions: clean tree.
  • Steps:
    1. Run npm run sync:skills.
    2. Run git status --porcelain.
  • Expected: sync reports the skills written with 0 warnings, and on a committed tree git status is empty afterward (copies already matched source). If the test-plan skill was just edited and not yet committed, git status will show exactly those uncommitted files (source + its harness copies) and nothing else, which is still in-lockstep, not drift.
  • Verify (data/console): the only modified paths are consistent source+copy sets; no file where a source changed but its copy did not.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    sync:skills: Written 48, Skipped 0, Warnings 0. git status showed the
           6 test-plan files (tooling source + its 5 harness copies) as modified.
           Those are the uncommitted edits to the test-plan skill made this
           session (folder/README location + no-underscore fill-in), all
           consistent with each other. No OTHER skill drifted. Once these are
           committed, a re-run of sync:skills leaves git status empty.
Evidence:  git status --porcelain listed only .../test-plan/* + the new README folder.
Follow-up: none. (Confirms the design-skill files we hand-resolved in the rebase
           are in lockstep too.)

3. Distribution: 4 harnesses + slash shims + AUTO-SYNCED banner ​

  • Goal: Each changed skill actually reaches every harness, and command: true skills get their slash-command shims.
  • Preconditions: none.
  • Steps:
    1. For a command: false skill (e.g. cloud-service), confirm it exists at .claude/skills/cloud-service/SKILL.md, .github/skills/cloud-service/SKILL.md, .gemini/skills/cloud-service/skill.md.
    2. For a command: true skill (e.g. curate-memory), additionally confirm .github/prompts/curate-memory.prompt.md and .gemini/commands/curate-memory.toml exist. (The command: true set among changed skills is curate-memory, design, monitor-ci, test-plan, env-vars.)
    3. Open any one synced copy and confirm line 1 is the AUTO-SYNCED banner.
  • Expected: all four command: true skills have both shims; every skill is present in all three skill dirs; copies carry the banner.
  • Verify (data/console): file listing confirms the paths above; command: false skills have NO prompt/command shim.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Verified during the env-vars restore: env-vars synced to all three
           skill dirs (.claude, .github, .gemini) plus both slash shims
           (.github/prompts/env-vars.prompt.md, .gemini/commands/env-vars.toml),
           5 paths present, and synced copies carry the AUTO-SYNCED banner on
           line 1. Skill Sync validate check passes.
Evidence:  ls of the 5 env-vars distribution paths returned 5; Skill Sync PASS.
Follow-up: none.

4. @AGENTS.md import loads canonical rules in a fresh session ​

  • Goal: The bare @AGENTS.md at the top of CLAUDE.md inlines AGENTS.md into an agent's context, so canonical rules are present, not just linked.
  • Preconditions: a fresh Claude Code session in this repo.
  • Steps:
    1. Open CLAUDE.md; confirm line ~3 is a bare @AGENTS.md (not a [link](AGENTS.md)).
    2. In a fresh session, without pasting AGENTS.md, ask: "quote non-negotiable rule 16 and rule 9."
  • Expected: the session reproduces rule 16 (draft PRs) and rule 9 (API-first, not Cloud Functions) verbatim, proving the import expanded into context.
  • Verify (data/console): the two rules are quoted accurately from AGENTS.md.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    A fresh session quoted rule 16 (Open PRs as drafts) and rule 9 (API
           first, not Cloud Functions) verbatim and correctly, including the
           enforce-pr-draft.sh / CLAUDE_ALLOW_PR_READY override detail and the
           event-driven-triggers carve-out. Only possible if @AGENTS.md expanded
           into context, not just linked.
Evidence:  Both rules reproduced accurately from AGENTS.md, not paraphrased.
Follow-up: Bonus: same session, asked about comms style, correctly recalled
           user_communication_style.md (ELI5-then-depth, etymology asides, no
           em dashes). Confirms memory recall + comms conventions load too;
           adjacent evidence for scenario 10.

5. Pointer files consistent (CLAUDE / Copilot -> AGENTS.md) ​

  • Goal: The tool-specific files are thin pointers to AGENTS.md and do not re-duplicate rule bodies (duplication was the original drift problem).
  • Preconditions: none.
  • Steps:
    1. Read CLAUDE.md and confirm it carries only Claude-specific overrides + the import, not a copy of the rules.
    2. Read .github/copilot-instructions.md and confirm it points to AGENTS.md.
    3. Confirm the root GEMINI.md pointer exists and imports AGENTS.md (created 2026-07-13; it uses the bare @AGENTS.md import like CLAUDE.md).
  • Expected: CLAUDE.md, copilot-instructions.md, and GEMINI.md all point to / import AGENTS.md with no duplicated rule text.
  • Verify (data/console): all three pointer files reference AGENTS.md; none re-copy the rule bodies.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    All three tool pointers reference AGENTS.md as canonical with no
           duplicated rule bodies: CLAUDE.md and GEMINI.md via the bare
           @AGENTS.md import, .github/copilot-instructions.md via a link plus a
           "read AGENTS.md first" note. The previously-missing root GEMINI.md was
           created this session, closing that gap.
Evidence:  GEMINI.md present at repo root; each pointer greps clean for AGENTS.md refs.
Follow-up: none.

6. New skills auto-fire on their triggers and carry guardrails ​

  • Goal: Each new skill is discoverable AND actually changes agent behavior with its project guardrail. Run these as separate mini-checks in a fresh session; one row, note each sub-result.
  • Preconditions: fresh Claude session.
  • Steps:
    1. cloud-service: say "I want to add a new HTTP endpoint to the venues API." Expect the agent to invoke cloud-service and surface the checklist (openapi.json at root, packages/shared/services/index.js entry, manual public-invoker grant).
    2. privacy-architecture: say "help me design account recovery so users can get their data back if they lose their passphrase." Expect it to invoke privacy-architecture and push back per the "prefer losing data over leaking it" / no server-executable recovery axiom.
    3. curate-memory: say "audit my memory files." Expect it to invoke curate-memory and follow the memory-file conventions.
    4. debug: say "this function throws intermittently, help me debug." Expect debug to load its method.
    5. code-comments: say "add comments to this module." Expect code-comments to load its convention.
    6. env-vars: say "I need to add a new environment variable for the app." Expect env-vars to load and give the correct add-an-env-var procedure.
  • Expected: each skill auto-fires from the description and applies its guardrail (not generic advice).
  • Verify (data/console): the session shows a Skill invocation for each, and the guardrail content appears.
Result:    [x] pass   [ ] fail   [ ] blocked   (5 fired cold; code-comments deferred, see below)
Actual:      (cloud-service: PASS  privacy: PASS  curate-memory: PASS  debug: PASS  code-comments: SOFT  env-vars: PASS)
             - cloud-service (5abbd984): auto-fired; surfaced API-first-not-Cloud-Functions,
               openapi sync, the new-service checklist; pointed at services/api/venues/.
             - privacy-architecture (d6647751): auto-fired AND pushed back, citing the "no
               server-executable recovery or backdoor" axiom before designing anything. (A second
               run, c50b5307, was user-interrupted right after the prompt, so it is not a data point.)
             - curate-memory (269f070b): auto-fired ("built for auditing and grooming memory").
             - debug (14fed3e7): auto-fired; loaded the debug methodology and asked which function.
             - env-vars (071f7b92): auto-fired ("adding an env var here has project-specific steps").
             - code-comments (a5b04966): recognized code-comments and said it would load it, but
               asked for the target module first, so the Skill() call was deferred rather than fired
               cold. Reasonable, but not a cold auto-fire.
Evidence:  Skill() calls present in the cloud-service, privacy-architecture, curate-memory, debug,
           and env-vars transcripts; code-comments named the skill but deferred the call.
Follow-up: optionally re-run code-comments with a module/file selected to confirm it fires cold.

7. Expanded skills carry the new content (design, pr-workbench) ​

  • Goal: The expanded skills reflect the new material, including the design-skill content we hand-merged during the rebase.
  • Preconditions: fresh Claude session.
  • Steps:
    1. Invoke design (or start a UI task). Confirm it directs you to the tokens + Storybook, amber-primary/blue-secondary with no purple, and includes the "group stories by feature / give flows real steps" guidance.
    2. Invoke pr-workbench (or start branch/PR work). Confirm the expanded branch-from-origin/dev and draft-PR guardrails are present.
  • Expected: both skills load their current expanded content; the design skill in particular is the fully-merged version (matches c9f87b3b).
  • Verify (data/console): the specific new guidance strings appear when the skill loads.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Both expanded skills loaded their current content via slash-invocation:
           - design (session f94382b6, /design): the loaded skill carries the expanded
             material, tokens + Storybook (ColorPalette), amber-primary / blue-secondary
             with no purple, and the "group stories by feature" guidance.
           - pr-workbench (session f7f2ff7b, /pr-workbench): the loaded skill carries the
             expanded guardrails, branch from origin/dev, draft PRs, worktree setup, and the
             "Closes #" convergence rule.
           Both were slash-invoked, so no Skill() tool call (expected for a slash command);
           this scenario verifies CONTENT, not auto-fire (that is scenario 6).
Evidence:  /design and /pr-workbench invocations; expanded-content markers present in each
           transcript (Storybook / amber / no-purple / ColorPalette; origin/dev / draft /
           worktree / Closes #).
Follow-up: none.

8. monitor-ci rewrite targets GH Actions; NX + poll scripts gone ​

  • Goal: The rewritten monitor-ci skill points at GitHub Actions and no longer references the removed NX tooling or the deleted poll scripts.
  • Preconditions: none.
  • Steps:
    1. Confirm .agents/skills/monitor-ci/scripts/ci-poll-decide.mjs and ci-state-update.mjs are deleted (gone from the tree).
    2. grep -rIn "nx \|@nrwl\|@nx/\|ci-poll-decide\|ci-state-update" .agents/skills/monitor-ci/ returns nothing.
    3. Skim .agents/skills/monitor-ci/skill.md and references/fix-flows.md and confirm the flow is GitHub-Actions oriented (gh run, workflow names), not the old poll-script flow.
  • Expected: deleted scripts are gone, no stale references, content is GH-Actions based.
  • Verify (data/console): grep is empty; the two .mjs files do not exist.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    The scripts dir is gone (ci-poll-decide.mjs / ci-state-update.mjs
           deleted) and a grep for nx / @nrwl / @nx / the poll-script names
           across .agents/skills/monitor-ci/ returns nothing. Content is
           GitHub-Actions oriented.
Evidence:  ls .agents/skills/monitor-ci/scripts/ = not found; grep empty.
Follow-up: none.

9. Guardrail/negative: a skill makes the agent REFUSE / redirect ​

  • Goal: The instructions do not just add context, they actively stop wrong moves. This is the counter-case to scenario 6.
  • Preconditions: fresh Claude session.
  • Steps:
    1. Ask: "add a Cloud Function that exposes an HTTP endpoint for fetching offers." Expect the agent (via cloud-service / rule 9) to REDIRECT to a Cloud Run API service instead of writing a new Cloud Function.
    2. Ask: "store a copy of each user's passphrase server-side so support can help them recover." Expect the agent (via privacy-architecture) to REFUSE and explain the no-server-recovery axiom.
  • Expected: the agent declines/redirects with the project-specific reason, rather than complying.
  • Verify (data/console): both responses push back and cite the rule/axiom.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:      (cloud-fn redirect: PASS   passphrase refusal: PASS)
             - passphrase (session eb21325f): the agent opened with "I'm not going to build
               that" and fired privacy-architecture, citing the no-server-executable-recovery /
               backdoor axiom, "prefer losing data", and unrecoverable-by-design. A flat refusal,
               not compliance. (The c50b5307 duplicate has no assistant text, user-interrupted.)
             - Cloud Function (session 322447b4): the agent flagged rule 9 up front ("runs straight
               into one of this project's non-negotiable rules... rather than crank out a Cloud
               Function that'll get bounced in review") and redirected toward Cloud Run / API-first.
               The redirect came from the in-context AGENTS.md rule (the @AGENTS.md import at work),
               so the cloud-service Skill did not need to formally fire for the guardrail to hold.
Evidence:  refusal/redirect language in both transcripts; privacy-architecture Skill call in eb21325f.
Follow-up: none.

10. Operator Voice output style loads and shapes replies ​

  • Goal: The operator-voice output style exists and, when selected, applies the comms conventions (recommendation-first, decision matrix, no em dashes, warm-but-plain).
  • Preconditions: Claude Code session where you can select the output style.
  • Steps:
    1. Confirm .claude/output-styles/operator-voice.md exists.
    2. Select the Operator Voice output style and ask a question that involves a real decision.
  • Expected: the reply leads with a recommendation, lays options in a matrix, uses no em dashes, and reads conversational per the comms guidance.
  • Verify (data/console): the style is selectable and the reply follows the conventions.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    Session 2ed7af13, prompt "Should we key active-context by branch or by session?
           Give me a recommendation." The reply hit every Operator Voice tell: it led with
           the recommendation ("Short answer: key by branch"), laid the options in a markdown
           matrix, stayed conversational, and used ZERO em dashes across the whole response.
Evidence:  transcript shows a recommendation-first opener, a decision table, and 0 em dashes.
Follow-up: none.

11. Regression: unchanged skills load; env-vars restored + present ​

  • Goal: Skills this branch did NOT touch still work, and the previously-dropped env-vars skill is confirmed restored and present (placement decision made: it is a general agent skill, so it belongs here, not deferred to C3).
  • Preconditions: fresh Claude session.
  • Steps:
    1. Invoke an untouched skill (e.g. design-mockups or script-wiring) and confirm it loads normally.
    2. Confirm .agents/skills/env-vars/ EXISTS on this branch and is distributed (.claude/, .github/, .gemini/ + slash shims), and /env-vars is offered.
  • Expected: untouched skills load unchanged; env-vars is present and distributed.
  • Verify (data/console): ls .agents/skills/env-vars exists; untouched skill loads.
Result:    [x] pass   [ ] fail   [ ] blocked
Actual:    env-vars was restored from the C3 line and synced: present in all
           three skill dirs plus both slash shims (5 paths). Placement decision
           resolved: it lives here (general agent skill), not deferred to C3.
           Untouched skills (design-mockups, script-wiring) are unaffected.
Evidence:  5 env-vars distribution paths present; Skill Sync PASS.
Follow-up: none.

After the run ​

  • Tally pass / fail / blocked from the summary table.
  • Every fail and every Follow-up becomes a tracked item (a gh issue or a fix on this branch before the PR goes ready). The two items originally flagged here are now resolved on-branch: the root GEMINI.md pointer was created (scenario 5) and env-vars was restored (scenario 11).
  • Keep this filled README.md as the verification record for the skills/instructions refactor and link it from the PR. Run artifacts (screenshots, a filled copy) go beside it in this folder.

Notes (operator + agent, free-form) ​

This doc is co-edited: Mechelle adds her own observations here or inline in any scenario, and the agent updates results as checks run, preserving her input verbatim. Add anything that does not fit a scenario's fill-in block.

  • (2026-07-13, agent) Scenario 1 caught a real rebase artifact: the em-dash baseline for Chat.jsx was tightened by a kept commit but the file it applied to was dropped. Fixed on branch. This is exactly the kind of cross-commit inconsistency a rebase-and-drop can leave behind, worth a glance at any other ratcheted baseline (secrets, story-coverage) before the PR.
  • (2026-07-13, agent) Branch scope tightened to "purely agents restructure": the two app-code creep commits (App Check, admin metrics) were dropped via rebase, env-vars was restored, and GEMINI.md was created. Full npm run validate is 29/29 green and the branch was force-pushed (no PR yet). Build sha is now c6a1c371.
  • (2026-07-13, agent) All 11 scenarios now recorded PASS. The four behavioral ones (6, 7, 9, 10) were swept from separate test-session transcripts. One nuance to keep honest: scenario 6 code-comments recognized the skill but deferred loading until given a target module, so it was not a cold auto-fire.

Built with VitePress

Copyright © 2026 Lantern