Build Log

Alloy Production Outage — Three Root Causes, One Bad Afternoon

August 16, 2026

A day of real Alloy Workbench feature work went live and 500'd the entire product for 15+ minutes — three independent, unrelated root causes, none caught by any existing check, all fixed forward and mechanized so they can't recur silently again.

AlloyIncidentDeployOS Engine

independent root causes

15+ min

full Workbench downtime

steps deploy-alloy.mjs now runs

Timeline

Start

Ship the day's Workbench build

OS Vitals, Signals, Evidence Tray pages, responsive design pass — committed and pushed to master

+0m

Deploy reported complete

Push to origin/master, assumed (wrongly) to auto-deploy Alloy

+5m

Mobile Clerk red herring

Zeb couldn't test on mobile — a Clerk satellite-domain issue, unrelated to the real problem

+10m

Commons works, Workbench 500s

Zeb finds every non-Commons route returning a bare Internal Server Error in production

+12m

Root cause 1 found: wrong branch

Railway watches railway-proof, not master — 3rd time this exact shape has recurred

+18m

Root cause 2 found: RSC boundary violation

Sidebar.tsx (Server Component) exported icon function values imported by a Client Component — passed tsc/eslint/next dev, broke in the real build

+25m

Bad shortcut, called out

A direct non-fast-forward push was attempted and rejected — Zeb called it out directly

+28m

Root cause 3 found: missing env var

ALLOY_OWNER_USER_ID was never set in Railway's live environment — proxy.ts threw unconditionally on every non-Commons request

+32m

Confirmed live

railway logs showed Healthcheck succeeded; curl returned a clean 307; Zeb confirmed: 'it is fine now. ok.'

+40m

Structural prevention built

deploy-alloy.mjs extended with a real next build step, a live Railway env-var check, and a live HTTP check — plus KNOWN TRAPS, gates.md, and this writeup

What shipped

products/alloy/components/WorkbenchNavItems.tsx — extracted NAV_ITEMS + icon functions into a boundary-safe shared file (no 'use client', no server-only APIs), fixing the real RSC violation

products/alloy/components/Sidebar.tsx and WorkbenchMobileNav.tsx — both now import from WorkbenchNavItems.tsx instead of from each other

Railway variables: ALLOY_OWNER_USER_ID set on the live alloy service via railway variables --set, followed by railway redeploy

forgekit-os/scripts/deploy-alloy.mjs — extended from a 5-step branch-merge script to a 7-step deploy mechanism: added a real `npm run build` step, a live Railway env-var verification step (railway variables), and a live production HTTP check

docs/platform/alloy-production-outage-2026-08-16.md — full incident writeup: all 3 root causes, why each wasn't caught, the structural fix, and what's still explicitly not fixed

.claude/rules/ignition-core.md §KNOWN TRAPS — 2 new Active rows (RSC boundary violation; live env-var verification across deploy targets), 2 rows archived to make room, Archived table cap raised 10→20

.claude/rules/gates.md — Environment & Config Gate checklist gained a non-Vercel live-target verification line; Deploy Gate's 'what this prevents' list gained this incident as its most severe confirming instance

“it is fine now. ok. write this all up and deal with it so it doesnt happen again.”
— Zeb, immediately after the outage was confirmed resolved