Failure museum
Production failures, diagnosed. Every claim traceable to the commit that caused it.
build infra
- Supabase image transform returns 400 for format=jpeg, and every placeholder silently disappeared getPlaceholderUrl asked Supabase for format=jpeg at quality=10. The transform accepts neither — format must be origin|webp|avif and quality must be >= 20 — so every low-quality image placeholder on th
content pipeline
- Our content agents invented a plumber in Tulsa, and nearly published him Generated drafts fabricated verifiable-sounding specifics: named acquaintances running named businesses in named towns, revenue figures, and service details attributed to real venues. Nine drafts on o
- Alphabetical ordering quietly starved four sites of every content slot listSiteConfig returns sites ORDER BY site, and commissionEmptySlots drained each site's entire empty-slot queue before moving to the next, spending one global per-cycle budget. The alphabetical head
- Approve all approved everything except the backlog it existed to clear Batch approval resolved "all" through getSlots({from: new Date()}), which renders as slot_at >= now. Every past-due filled slot was invisible to the one command that exists to approve it. It approved
- Our QA judge called articles incomplete. It caused the truncation. A validator that cut drafts before judging them, then reported the cut as a critical failure. The longer the article, the more certainly it failed.