INC-24ops
A content pipeline billed about 64 million Opus tokens to the metered API because Bun auto-loaded a .env
The page pipeline spawned `claude -p` as a child process and passed it the parent environment unchanged.
Failure museum
Production failures, diagnosed. Every claim traceable to the commit that caused it.
Cluster 01
ops5 published
- INC-21ops
One account-security change invalidated the Meta login, and every Facebook page and Instagram account disconnected at once
On 2026-10-19 a change to the sign-in security on the owner's Meta login led Meta to invalidate the session, and every connected Facebook page token failed with HTTP 401 (190/460) "The session has bee
- INC-16ops
We added random jitter to every queued email. With enough rows waiting it cancelled itself out.
Cold outreach paced sends by giving each waiting row its own random delay of up to 20 minutes, on top of a 6-minute minimum gap.
- INC-10ops
Cloudflare's "Add a site" DNS import proxied the mail CNAMEs and left two copies of the SPF record
Migrating two domains that carried live mail into our Cloudflare account on 2026-07-29, we found the "Add a site" DNS scan unsafe to trust.
- INC-08ops
One memory cap killed all four interactive sessions at once: they all shared the bot's cgroup
On 2026-07-27 a memory-cgroup OOM killed the three numbered work sessions and the bot's own session together.
Cluster 02
agent behaviour5 published
- INC-23agent-behaviour
Four of the owner's decisions never reached the agent session that had asked for them
Only one of the studio's Claude Code sessions is bridged to Discord, so when a session asked the owner a question, his reply arrived at a different session.
- INC-18agent-behaviour
Killing every `bun` process by name to stop a hung test took down 13 services that systemd would not restart
A test-running subagent piped `ps` into `xargs kill` against every `bun` process to stop a hung browser test on 2026-10-15.
- INC-14agent-behaviour
Our voice assistant asked for the same confirmation 3-4 times because its move gate had no anti-loop guard
Moving blockers to "resolved" by voice needed 3-4 repeated "yes" answers per batch.
- INC-13agent-behaviour
git add -A in a shared checkout committed 27 files of another agent's unfinished work to main
While syncing a small fix into a checkout that several agent sessions share, one session ran `git add -A`.
- INC-12agent-behaviour
Work the agent delivered stayed on the to-do list forever, because closing the request never closed the to-do
When the owner asks the voice assistant to "get Claude to do X", it writes an escalation-queue entry and, seconds later in the same turn, the capture path mints a to-do for the same X.
Cluster 03
content pipeline7 published
- INC-22content-pipeline
A museum evidence ref pointed at HEAD. When the code moved on, it stopped every later incident from being checked.
One incident's evidence pack had a file reference with no pinned revision, so it resolved against HEAD. Once the code was fixed, that line was no longer the fault.
- INC-20content-pipeline
Our menu extractor put "not vegetarian" in the field for diets a dish is suitable for
The menu data pipeline asked the model for one field, diet_inferred, holding what it inferred.
- INC-09content-pipeline
The default API URL in our content agent pointed at a host Cloudflare blocks for programmatic clients
The content agent defaulted SCRIPTGRAIN_BASE_URL to the ScriptGrain marketing domain's functions path.
- INC-04content-pipeline
Our content agents invented a plumber in Tulsa, and nearly published him
Generated drafts fabricated verifiable-sounding specifics: named acquaintances running named businesses in named towns, revenue figures, and service details attributed to real venues.
- INC-03content-pipeline
Alphabetical ordering quietly starved four sites of every content slot
listSiteConfig returns sites ORDER BY site, and commissionEmptySlots drained each site's entire empty-slot queue before moving to the next, spending one global per-cycle budget.
- INC-02content-pipeline
Approve all approved everything except the backlog it existed to clear
Batch approval resolved "all" through getSlots({from: new Date()}), which renders as slot_at >= now. Every past-due filled slot was invisible to the one command that exists to approve it.
- INC-01content-pipeline
Our QA judge called articles incomplete. It caused the truncation.
A validator that cut drafts before judging them, then reported the cut as a critical failure. The longer the article, the more certainly it failed.
Cluster 04
build infra4 published
- INC-19build-infra
Two branches both bumped a package to 1.42.0, git merged them without a conflict, and one release never shipped
One branch published a new street assembly as 3d-kit 1.42.0 and merged to main. A second release branch, cut earlier from the same parent, also bumped package.json from 1.41.0 to 1.42.0.
- INC-17build-infra
The vendored-library pin said one version and named the commit of another
Decksmade vendors the chart library and records the pin as a tag plus a commit SHA.
- INC-06build-infra
Deploying an edge function from an unmerged branch silently overwrote another session's live changes for 25 minutes
During a ScriptGrain long-drafts release, the live `api-v1` and `generate-content` edge functions were deployed from the feature branch before it was merged.
- INC-05build-infra
Supabase image transform returns 400 for format=jpeg, and every placeholder silently disappeared
getPlaceholderUrl asked Supabase for format=jpeg at quality=10.
Cluster 05
product1 published
- INC-15product
We gave a chart a fixed height, and its volume panes were added on top of it
In the chart library, spec.size.height on charts with stacked panes (candlestick volume and extra panes, and the stacked dual chart) set only the main pane's height.
Cluster 06
crawlability2 published
- INC-11crawlability
Cloudflare reported 10,247 visitors; our own cookieless analytics counted 282 sessions
Roadmap decisions were being steered by Cloudflare zone-analytics "unique visitors".
- INC-07crawlability
Our AI-recall test counted "mentioned" if the answer contained the phrase, but the question already contained the phrase
The AI-visibility canary probe plants made-up marker phrases and asks each assistant about them, scoring `mentioned` as whether the answer contains the marker.