Build Log

MedinaCHF Full Production Outage — the Dev/Prod Split Itself Became the Trap

August 16, 2026

Every authenticated page 500'd in real production because a local .env.production.local file quietly pointed at the wrong Neon branch — root-caused by cross-referencing Vercel's real env scope and Neon's real branch list, not by trusting either side's own claim.

MedinaCHFProduction IncidentSchemaEnvironment & ConfigDeploy

authenticated page types down in real production (member profile, directory, coordinator members, community)

Neon branches confused for one another (dev vs main)

missing columns applied directly to the real production branch

data rows disturbed (family count verified unchanged: 8)

Timeline

Start

Zeb reported a live production 500 on /coordinator/members with a screenshot showing an RSC render error

Read HOW_TO_WORK_WITH_ZEB.md + PROJECT_INDEX.md per ignition arc; Zeb clarified mid-report that multiple pages were affected (directory, events planner, My Profile) and that everything worked in dev

Misdirection

First hypothesis (missing Clerk publishableKey) was a self-inflicted artifact of an incomplete local repro, not the real cause

Ran a real `next build` + `next start` locally against `.env.production.local` and got a genuine 500 — but for the wrong reason (a NEXT_PUBLIC_* var not present at MY build time), which happened to produce a similarly-shaped but unrelated error. Zeb correctly called this out as guessing.

Correction

Zeb: 'stop guessing at shit... look at what we have recently done and promoted earlier today'

Redirected from local repro theorizing to actual git history and today's prior session retro (medinachf-home-address-2026-08-16), which had just added 5 new `families` columns and claimed to have verified them on prod

Ground truth

Pulled real Vercel production runtime logs and found the actual live error: NeonDbError column "home_street_address" does not exist (code 42703)

This was the first genuinely authoritative fact in the session — a real error from the real running Lambda, not a local reproduction or an inference

Contradiction

Directly querying `.env.production.local`'s own DATABASE_URL showed all 5 columns present — appearing to contradict the live error

Zeb supplied the missing fact directly: 'dev and preview are the same and production has its own database' — .env.production.local pointed at the wrong (dev) branch the whole time

Confirm

Cross-referenced Vercel's real Production-scoped env var (vercel env ls, not env pull — pull silently returns empty for sensitive vars) against Neon's actual branch list (neonctl branches list)

Found two branches with two different hostnames: main (ep-morning-mountain-..., 8 families, 0 home_* columns) and dev (ep-sparkling-boat-..., created 2026-08-15, has the columns). Root cause fully confirmed against authoritative sources on both sides.

Fix

Applied 5 ALTER TABLE ADD COLUMN IF NOT EXISTS statements directly to the real main branch via neonctl-obtained connection string, after explicit Zeb confirmation

Verified before/after column state and family row count (unchanged at 8) around the write. Confirmed zero new errors in live Vercel logs after the fix.

Close the loop

Zeb corrected .env.production.local to point at the real main branch

Independently re-verified: the file now resolves to ep-morning-mountain-... with all 5 columns present, matching live Vercel Production exactly

What shipped

No application code changed. This session's only durable changes: 5 columns added directly to the real production `families` table on Neon's `main` branch, and `.claude/rules/gates.md` §Environment & Config Gate updated with the incident record and a new checklist item.

.env.production.local (local machine only, gitignored) — corrected by Zeb to point at the real `main` production branch instead of the `dev` branch it had been pointing at since the split was created 2026-08-15

“you are guessing again. stop it”
— Zeb, mid-diagnosis, after a second local finding was presented before the real system's state was checked