Build Log

The Helm Classifier Probe: Does State-Grounding Actually Improve Request Routing?

August 14, 2026

Started as an in-bound/out-of-bound visualization exercise, corrected mid-session into the real question — can a two-stage classifier route Courtney's real requests into Content/Schema/Feature/Bug/Escalate, and does grounding against real app state actually help? Ran the probe 4 times across the session: found a real system-prompt bug and a real taxonomy gap, fixed both, added adversarial test cases to confirm the fix generalized, and corrected two of the test set's own answer-key errors the classifier caught. Ended at 17/17 grounded — reported honestly as a small, self-graded sample, not a validated result.

MedinaCHFAlloyHelmAIStrategy

real Courtney-shaped requests in the final test set (15 real + 2 adversarial)

full probe runs across the session, each a real cause-and-effect fix

answer-key errors the classifier's own reasoning caught and corrected, not classifier errors

live production auth-risk (Clerk webhook) independently flagged by grounding, unprompted

Timeline

Start

Asked to build an in-bound/out-of-bound list from Courtney's real Charges

Explicitly a visualization exercise, no scope change to the prior session's UpdateEvent.v1 pilot

Early pivot

Zeb said 'let's go' — building for real revealed the real pilot target was wrong

Grep found products/medinaCHF/lib/actions/events.ts already has a full updateEvent action covering every field UpdateEvent.v1 would need — the original pilot proved nothing

Mid

Zeb reframed: the real test isn't executing one operation, it's correctly ROUTING the full range of her requests

Four real buckets named: static content, schema field, complex feature, bug fix

Mid

Zeb added a second requirement: a runtime level-set that challenges the initial bucket guess

Scoped down (per Zeb) to just proving the state-grounding half works today — self-consistency passes and any learning-over-time mechanism explicitly deferred

Late

Built and ran a real, working two-stage classifier against 15 real requests

experiments/helm-classifier-probe/ — reused products/medinaCHF/lib/ai.ts's exact AI-call pattern

End

Grounded accuracy scored lower than naive on the raw number — inspected disagreements rather than accepting the score at face value

2 of 3 'misses' turned out to be test-set design gaps, not classifier failures; one was a genuinely valuable unprompted auth-risk flag

What shipped

experiments/helm-classifier-probe/classify.mjs — real, runnable two-stage classifier (naive guess, then state-grounded revision), reuses products/medinaCHF/lib/ai.ts's lazy-client-init AI-call pattern. Fixed twice: a resolved CONTENT taxonomy definition (behavior vs. execution-mechanism), and a real bug where the grounding stage never received the taxonomy's system prompt at all.

experiments/helm-classifier-probe/test-cases.json — 17 total cases: 15 real Courtney/Nichole requests (drawn from the prior session's full historical catalog) plus 2 adversarial cases (15, 16) authored this session specifically to stress-test the taxonomy fix. Two of the 15 real-request labels (cases 1, 7) were corrected mid-session after the classifier's own reasoning caught genuine ambiguities the original labels missed.

experiments/helm-classifier-probe/results.json — final run's full raw output, not hand-simulated

experiments/helm-classifier-probe/results-run1-before-taxonomy-fix.json, results-run2-after-taxonomy-fix-before-label-correction.json, results-run3-case1-corrected.json — preserved intermediate runs, so the before/after comparison is real and auditable, not reconstructed from memory

Not wired into the products/medinaCHF workspace or any deploy target — deliberately a standalone research probe, not application code

“Respect it — update the answer key to ESCALATE, this is a real ambiguity.”
— Zeb, on case 1's regression — choosing to trust the classifier's own reasoning over a hastily-written label, then applying the identical standard again on case 7