AI milestone classifier - Test Plan and Evaluation

Field	Value
Change	.github/workflows/assign-milestone.yml classifies new issues into Prototype / Alpha / Beta instead of stamping every issue Prototype
Branch / PR	folded into #917
Precedent followed	lint-pr-labels.yml AI fallback: claude-haiku-4-5-20251001, same ANTHROPIC_API_KEY, allowlist-validated, env-only untrusted input, graceful no-op
Correction layer	#923, a weekly janitor re-reviews these assignments
Date	2026-08-18, classifier commit 21:01 UTC (14:01 PT)

Summary

#	Scenario	Result
1	Accuracy against hand-sorted issues	[x] 78% (14/18)
2	Mechanical fallbacks all decline safely	[x] 4/4
3	Workflow shell runs end to end (dry-run)	[x] pass
4	Audit-trail comment identifies AI assignments	[x] pass
5	Confidence-based declining works	[x] NO, it does not
6	Live behaviour on a real new issue	[ ] pending the first real issue

1. Accuracy: 78% against a stratified labelled set

Scored with node --env-file=.env.local tooling/scripts/classify-milestone-dryrun.mjs, which pulls issues the operator had already sorted by hand and compares against her answers.

The sample is stratified deliberately. The 12 most recent issues are all Prototype, so a classifier that answered "Prototype" every time would score 12/12 and look perfect. Six per stage is what lets the score come back bad.

Expected stage	Agreed
Prototype	6/6
Alpha	4/6
Beta	4/6
Total	14/18 (78%)

Baseline: the unconditional Prototype stamp is right about 55% of the time given the real distribution (97 Prototype / 70 Alpha / 9 Beta). The classifier is a real improvement on what ships today.

EXPECTED-MISS CLASS: security or privacy work implemented as tooling

If a milestone looks wrong, check this pattern first. It is known and measured, not a new bug.

The stable errors are all one shape: an issue whose SUBJECT is security or privacy but whose MECHANISM is a lint rule, a CI gate, a guard or a refactor. The classifier reads the mechanism and files Prototype; the operator reads the subject and sorts Alpha.

Observed: #895 and #893, both labelled bug,security,devops, both hand-sorted Alpha, both classified Prototype. Also #631, hand-sorted Beta and classified Alpha, because a safety feature deferred pending Alpha evidence reads as Alpha-shaped.

A tie-break instructing "subject beats shape" was added to the prompt and improved Alpha recall from 3/6 to 4/6 without closing it. Both readings are genuinely defensible, which is why prompt wording does not fix it. #923 is the intended correction layer.

5. Confidence-based declining does NOT work

The most important negative result, and it changed the design.

The intended safety valve was "unsure means unmilestoned", because a wrong stage is worse than an empty one. Three implementations were tried and all three failed:

Valve	Result
The model's self-reported confidence	Reported "high" on all 18, including all 5 misses. Raising the bar to high-only changed nothing at all.
Labels plus a subject-beats-shape tie-break	Improved recall, drove declines from 2 to 0. The valve stopped firing exactly as the model grew more certain.
Self-consistency, two samples at temperature 1	Best score (78%) and still 0 declines. The samples agree with each other and are still wrong on the same issues.

The errors are systematic, not noisy. The model reasons to the same wrong answer every time, so no confidence signal separates right from wrong. A self-graded score cannot be the safety valve when the grader is the thing being checked.

What remains is the MECHANICAL fallback, which does work and is what ships.

2. Mechanical fallbacks (4/4)

Exercising the exact CLI contract the workflow uses:

Input	Result	Exit
Normal call, issue #886	{"milestone":"Alpha",...}	0
No ANTHROPIC_API_KEY	{"milestone":null,"reason":"ANTHROPIC_API_KEY not configured"}	0
Empty allowlist	{"milestone":null,"reason":"no milestones in the allowlist"}	0
No title	{"milestone":null,"reason":"issue has no title to classify"}	0

Always exit 0: a classifier failure must never fail the workflow.

3. Workflow shell, end to end

YAML parsing proves the file is well-formed and nothing about the shell inside it, so the run: block was extracted and executed locally in dry-run against a real issue.

Result: Classified as "Alpha", then DRY-RUN: would assign #886 to "Alpha", exit 0.

A bug this caught that YAML validation could not: classify_issue is called with $(...), so it runs in a subshell, and a variable set inside it does not survive. The reasoning was being passed that way and would have produced an empty reason in every audit-trail comment. It now travels via a file, verified non-empty (200 bytes) after the run.

Method: testing an event-driven workflow without polluting the tracker

Reusable, and the reason this change could be proven at all.

An issues: opened workflow normally needs a real new issue to exercise, which means test noise in the operator's tracker and a wait for the event. Worse here: every open issue already HAS a milestone, so the assignment path was unreachable even with a real issue to hand.

The approach that worked, and generalises to any workflow of this shape:

1. Extract the run: block from the YAML and execute it directly. YAML parsing and actionlint both prove the file is well-formed and neither executes a single line of the shell inside it, which is where the logic lives.
2. Feed it the event's env vars by hand (EVENT_NAME, ISSUE_NUMBER, ISSUE_TITLE, ISSUE_BODY), which is exactly what the workflow receives.
3. Put a gh shim first on PATH that proxies every call to the real gh except the single read you need to control. Here it answers only the existing-milestone probe as empty, so the issue looks unmilestoned.
4. Run with DRY_RUN=true, so the real classifier, the real API call and the real milestone lookup all execute while nothing is written.

That combination exercises the true code path against live data with zero writes and zero test issues. It is what caught the subshell bug described above, which every static check passed.

4. Audit trail

Every AI assignment posts a comment naming the stage, the reasoning, and the fact that a human correction will not be undone. This is what lets #923 correct AI assignments without ever touching a milestone a person set by hand.

6. Live behaviour (pending)

• Steps: file the next real issue normally, then check it received a plausible stage plus the audit-trail comment, or was left unmilestoned with a warning in the run log.
• Watch for: the expected-miss class above, before reporting a new bug.

Result:    [ ] pass   [ ] fail   [ ] blocked
Actual:
Evidence:

Scope deliberately not changed

PRs and the backfill path still use the unconditional Prototype stamp. The dispatch was about new issues; a PR's stage should probably follow its issue, which is a different question; and backfill is currently a no-op because every open issue already has a milestone. Changing either would have been unrequested scope on a workflow that writes to the operator's board.

Not established

• Any measurement of live behaviour. Every number here comes from replaying issues that already have answers.
• Whether 78% holds on future issues. The sample is 18, all from one already-sorted backlog, and prompt changes were evaluated against that same set, so some overfitting to it is likely.