Build Log

Self-Consistency Pass: A Real, Pre-Declared Negative Result

August 14, 2026

Designed and built an adversarial third stage for the Helm classifier — same grounded evidence, argue for a different bucket, escalate on real disagreement. Stated the success criterion before running it. It correctly caught both known ambiguities, but false-escalated 3 of 15 clear-cut cases by constructing technically-coherent-but-impractical counter-arguments. Recorded as a genuine negative result, not iterated into a passing score.

MedinaCHFHelmAIVerification

of 2 remaining design decisions from the classifier-probe session, chosen this session

2/2

known-ambiguous cases correctly escalated

3/15

clear-cut cases falsely escalated — success criterion NOT met

real API calls made across the 17-case run (3 stages x 17 cases)

Timeline

Start

Zeb: 'second pass' — chose the self-consistency design decision from the deconfound session's nextItems menu

The other named option (learning-over-time mechanism) was left untouched this session

Design

3 design questions resolved via AskUserQuestion before any code was written

Independence axis (adversarial framing, not sampling variance), disagreement handling (any real disagreement forces ESCALATE, no adjudication call), and a pre-declared success criterion

Build

Built a true additive third stage, reusing classify.mjs's taxonomy/grounding logic verbatim

classify-with-self-consistency.mjs — stage 3 argues for a DIFFERENT bucket using the same evidence, explicitly permitted to say no genuine argument exists

Run

Ran all 17 cases through all 3 stages — 51 real API calls

Correctly escalated both known-ambiguous cases (1, 7); also escalated 3 clear-cut cases (11, 14, 16) that the 2-stage classifier already got right

End

Recorded the negative result honestly, declined to iterate the prompt to force a passing score this session

Zeb: record it, stop here for now — a fresh future session decides what comes next

What shipped

experiments/helm-classifier-probe/classify-with-self-consistency.mjs — a real, additive third classification stage. Reuses classify.mjs's TAXONOMY/GROUNDING_PROMPT/BUCKETS verbatim (not re-derived) to stay a true addition on top of the fixed baseline, not a parallel reimplementation that could drift from it.

experiments/helm-classifier-probe/results-self-consistency.json — full real run output, including per-case adversarial reasoning and an explicit successCriterionMet: false field computed directly in the script, not asserted after the fact

“Record it as a real negative result, stop here for now.”
— Zeb, declining to iterate the adversarial prompt to force a passing score on the same 17-case test set