Build Log

De-confounding the Helm Classifier's Two Simultaneous Fixes

August 14, 2026

The prior session's crucible:wrap review flagged that two real fixes (a missing system prompt, an ambiguous taxonomy definition) were applied and measured together, with their individual contributions never isolated. This session built two isolated variants to answer which fix actually mattered — the answer was clear and not the assumed 50/50 split.

MedinaCHFHelmAIVerification

isolated variants built and run

17/17

variant A (system-prompt fix present, taxonomy text still ambiguous) accuracy

14/17

variant B (system prompt absent, taxonomy text fixed in script only) accuracy

fix confirmed as the dominant, load-bearing cause

Timeline

Start

Zeb: 'let's continue to work' — presented the 3 named next steps from the prior retro

Zeb chose de-confounding over the two design-only options, as the cheapest way to close a real evidentiary gap

Mid

Built 2 isolated variant scripts, each reverting exactly one of the two prior fixes

Variant A: system-prompt fix kept, taxonomy text reverted to original ambiguous wording. Variant B: taxonomy fix kept, system-prompt bug deliberately reintroduced

Mid

Ran both against the same 17-case set

Variant A: 17/17. Variant B: 14/17, reproducing the exact same case-3 mistake as the original broken run

End

Concluded the system-prompt fix was necessary; the taxonomy fix's marginal contribution is unproven by this data

Confirmed with Zeb before writing it up this way rather than defaulting to 'both fixes mattered equally'

What shipped

experiments/helm-classifier-probe/classify-variant-a-systemprompt-fix-only.mjs — system prompt present in both stages, CONTENT taxonomy text reverted verbatim to its original, pre-fix ambiguous wording

experiments/helm-classifier-probe/classify-variant-b-taxonomy-fix-only.mjs — CONTENT taxonomy text kept at its fixed, resolved wording, but the grounding stage's system prompt deliberately omitted again (reintroducing the original bug on purpose)

experiments/helm-classifier-probe/results-variant-a.json, results-variant-b.json — real run outputs from both isolated variants

Both variant scripts are throwaway research instruments, explicitly marked for deletion once this question is answered and recorded — not meant to be maintained alongside the real classify.mjs

“The system-prompt fix was necessary; the taxonomy fix's marginal contribution is unproven.”
— Zeb, choosing the precise, narrower framing over either overclaiming a 50/50 split or overcorrecting to 'the taxonomy fix was pointless'