A museum evidence ref pointed at HEAD. When the code moved on, it stopped every later incident from being checked.

An unpinned ref on editor-commission-budget-starvation moved at HEAD

A reference that points at HEAD is a promise that the thing it points at will never change. Ours broke that promise the moment we fixed the bug it was meant to evidence. And then it took the whole weekly run down with it.

Symptom

The weekly run stopped part way through. No later incident got verified. That's the thing I noticed first, not the error itself: the run just didn't finish, and everything queued behind the failing item sat there untouched.

The line in the output was this, exactly:

An unpinned ref on editor-commission-budget-starvation moved at HEAD

That's the verifier talking. It had looked at a code reference in the evidence pack for that incident, resolved it, and decided the line no longer showed what the post said it showed. Fair enough. Then the tool tried to record that verdict by writing evidence_ok=false against the post, and the post was already published. A database trigger refuses that write. The exception went up the stack and nothing caught it, so the run just ended.

So two failures, really. One was a stale reference. The other was a crash that took innocent incidents down with it.

What we thought it was

My first guess was the trigger. It's the loud part of the failure, the part with an exception attached, and it lives in src/db.ts, which is where you go when a write gets refused. A trigger that blocks updates on published posts is doing its job, sure, but a trigger blocking something it shouldn't is a very believable bug. I spent a while reading it, looking for a condition that was too broad.

It wasn't. The trigger was right. A published post is supposed to be frozen; that's the whole point of it, and the refusal was correct behaviour. I'd have loved the bug to be there because then it's a one-line change. Anyway, that theory died fairly quickly once I read what the trigger actually guards.

The second guess was the verifier being flaky. The string says a ref "moved", and movement sounds like something non-deterministic, like a resolver returning different answers on different runs. Also believable. But the verifier was deterministic and doing exactly what it should. Something had genuinely moved, and the verifier was right to notice.

Root cause

The evidence pack for that incident held a file reference with no pinned revision. In tools/museum-seed.ts the reference at line 140 was written without a commit attached, so it resolved against HEAD every time anyone looked at it.

For a while that was fine. HEAD showed the faulty code, the code reference showed the fault, and the post's claim held up. Then the code was fixed, in commit 1dab1a6. From that point the line at HEAD was no longer the fault. The reference hadn't changed at all; the thing it pointed at had. That's the nasty part of an unpinned ref: it can go from true to false without a single edit to the document that contains it.

Next, the verifier. In tools/museum-verify.ts, line 65 checks the reference against what it resolves to now and fails it when they disagree. Correct, and it produced the string above. The failure then needs somewhere to go, so the tool attempted to set evidence_ok=false on the post.

And the post was published. The trigger refused the update, which is the behaviour you want from the trigger, and the exception propagated out of the loop because nothing in the loop expected a write to be refused. One regressed post ended the run for everyone after it. That's the crash, and it's a separate bug from the stale reference even though the stale reference set it off.

The fix

Two changes, matching the two failures.

First, the reference itself. We pinned it to the commit before the fix, which is the state of the code when the fault was real. The reference now reads as tools/museum-seed.ts:140@1dab1a6^, the parent of the fixing commit, so it resolves to the same lines regardless of where HEAD goes. The pinned form of the verifier line works the same way, tools/museum-verify.ts:65@1dab1a6^, and the database one is src/db.ts:70@23ca593^. Same idea each time: name the revision, and add the caret so you land on the parent of the change rather than the change itself.

Second, the run. A refused write on a published post is now treated as data rather than as an emergency. The tool reports the regressed post, carries on with the rest of the incidents, and raises a desk item from the weekly run so a person decides what to do about a published post whose evidence no longer holds. The trigger stays exactly as strict as before. We did not loosen it. The fix landed on 2026-10-21.

Why it happened

Nobody reviewing the evidence pack could see the problem, because the problem wasn't in the evidence pack. A reference without a revision looks complete. It has a path, it has a line number, and on the day you review it, it resolves to exactly the right thing. Review checks that a reference is correct today. It has no natural way of asking whether the reference will still be correct after the next commit.

The error handling had the same blind spot from the other direction. The loop was written for the case where verification fails and the write succeeds. A failed verification on an already published post was a combination nobody had walked through, and an unhandled exception is what an unconsidered combination looks like.

How to avoid it

If you cite code as evidence, pin it. A reference with no revision is a claim about a moving target, and the only question is when it stops being true. Anchor it to the commit where the thing was real.

And write batch jobs so one bad item can't veto the rest. Catch the refusal at the item level, record it somewhere a human will see, and keep going. A guard that blocks a write is working; the code that triggered the write should expect to be told no.