Build Log

Self-Consistency v2: Clean 21/21, Success Criterion Met

August 14, 2026

Following the exact next action named in the v1 negative result: a stricter adversarial prompt (reasonableness framing plus a rebuttal-question checkpoint), tested against the original 17 cases plus 4 new held-out clear-cut cases. Zero false escalations, both known ambiguities correctly caught, success criterion met on the first real run.

MedinaCHFHelmAIVerification

21/21

final (3-stage) accuracy across original + held-out cases

2/2

known-ambiguous cases correctly escalated

0/19

clear-cut cases falsely escalated (0 across all 15 original + 4 held-out)

MET

pre-declared success criterion — first real run, no tuning

Timeline

Start

Zeb: 'proceed with both items in order' — Option A first, per its own named next action

The v1 retro's nextItems specified exactly what v2 needed to do; this session executed that plan, not a fresh design

Design

One remaining design choice confirmed via AskUserQuestion: how to fix v1's specific failure

Combined both proposed mechanisms — a reasonableness framing plus an explicit rebuttal-question checkpoint — rather than either alone

Build

Authored 4 new, independently code-verified held-out cases before writing the v2 prompt

All 4 grounded in real grep/read evidence (announcements.ts, family-profile.ts, member-management.ts) — not invented, and deliberately not used to shape the prompt

Run

One real run hit a genuine process stall (killed and restarted once) before completing cleanly

21 cases, 63 real API calls, ~30+ minutes wall-clock — confirmed via process inspection (9+ min CPU time on one PID) before killing, not assumed

End

21/21, criterion MET — but one case's reasoning was visibly hedgier than the rest, named honestly rather than smoothed over

Case 16 (add-new-nav-link) reached the right bucket while its own rebuttal check called the classification 'imperfect'

What shipped

experiments/helm-classifier-probe/classify-with-self-consistency-v2.mjs — the fixed adversarial pass: reasonableness framing ('a technically-constructible argument is not enough') plus an explicit REBUTTAL_CHECK the model must answer before it's allowed to output CAN_ARGUE_DIFFERENT: YES

experiments/helm-classifier-probe/held-out-cases.json — 4 new, independently code-verified clear-cut test cases (all CONTENT-bucket — see the honest coverage-gap note below), not used to design the v2 prompt

experiments/helm-classifier-probe/results-self-consistency-v2.json — full real run output including every case's REBUTTAL_CHECK reasoning, not just the final bucket

“A real engineer who got this request would genuinely pause and ask 'does she want me to flip one event's status, or does she want a cap enforced going forward?' — these are materially different tasks and the request truly doesn't say.”
— The v2 classifier's own rebuttal-check reasoning on case 1 — the kind of grounded judgment the reasonableness framing was designed to produce