INC-17build-infra

The vendored-library pin said one version and named the commit of another

Decksmade vendors the chart library and records the pin as a tag plus a commit SHA. The tag said v1.3.6, but the SHA still named the commit behind v1.2.0, so the provenance record described code that

Verbatim symptom

charts.sha still named the v1.2.0 commit while ref and vendor were v1.3.6

# The Pin That Said Two Things

Here's a provenance record that told the truth about a version and lied about the code. Both at once, in the same file. I only noticed because two lines that should have agreed were sitting right next to each other, and they didn't.

Symptom

Decksmade vendors its chart library. The pin is recorded as a tag plus a commit SHA, which is a perfectly reasonable thing to do, you know, belt and braces. The tag tells a human what version they're looking at, and the SHA tells a machine exactly which tree.

So I was reading the provenance record, and what it said was this: charts.sha still named the v1.2.0 commit while ref and vendor were v1.3.6.

Read that again. Two of the three fields said v1.3.6. One said v1.2.0. And the one that said v1.2.0 was the one that's supposed to be unarguable.

Nothing crashed. Nothing paged anyone. That's what made it unpleasant, actually. The record simply described code that was not the code shipped, and it did so quietly, in a field nobody had a reason to distrust.

What we thought it was

My first diagnosis was a stale vendoring step. The theory went like this: somebody re-vendored the library at the newer tag, the files updated, but the step that writes the SHA didn't re-run, or ran against a cached checkout. A stale write, in other words.

It was believable, and I'll say why. Vendoring flows really do have a copy step and a record step, and those two can drift apart. If the record step reads from somewhere other than the files it just copied, you get exactly this shape: new content, old metadata. I've seen that bug in other forms.

I spent a while chasing it. I looked for a cache, a second checkout, anything that could hand the record step an old revision. There wasn't one. The theory was tidy and wrong, which is the worst kind, because tidy theories make you stop looking.

Anyway, the thing that broke it was a simple question. Where does the SHA come from? Not "why is it stale", just where does it come from.

Root cause

It doesn't come from anywhere. That's the answer. Nothing derives it.

The pin lives in package.json, at package.json:33@08e36ca^ and package.json:34@08e36ca^, which is the file as it stood before the fix. The tag had been bumped to v1.3.6. The SHA, sitting on the neighbouring line, had not. They were two hand-maintained values describing one thing, and somebody updated one and not the other.

That's it. No stale cache, no drifting step. A person changed a tag and forgot that a SHA existed beside it, and nothing in the repository was positioned to object.

Worth being precise here: the SHA wasn't wrong in the sense of being garbage. It was a real commit. It was the real commit behind v1.2.0. So it looked completely legitimate, which is exactly why it survived. A malformed hash gets caught; a valid hash for the wrong version walks straight through.

There was a second gap, related but separate, in deploy/worker-box/setup.sh:85@08e36ca^. The setup script works out which revision it's deploying. If you handed it a copy of the tree with no .git directory, there was nothing to ask, and instead of failing it recorded the revision as 'unknown'.

Think about what that does. Once 'unknown' is written down, the record is internally consistent and completely empty. After that, nobody could say what the box was running. We genuinely couldn't say, because the one place that should have held the answer held a placeholder.

So that's two different ways to end up with a provenance record that can't be trusted. One lies confidently. The other shrugs.

The fix

The fix landed as commit 08e36ca on 2026-10-21, and it covered both.

For the pin, the tag and the SHA now agree, so the record describes the code that actually ships. That part is almost boring, which is fine.

For the setup script, the behaviour changed in kind rather than degree. It no longer falls back to 'unknown'. When it's given a tree with no .git, it stops and asks for the revision to be given. Explicitly, by the person running it.

I like that more than I expected to. A script that guesses, or papers over a gap with a placeholder, is making a decision on your behalf and hiding that it did. A script that stops and says "I can't know this, tell me" pushes the decision back to someone who can actually make it.

Why it happened

Review didn't catch it, and I don't think that's because the reviewer was careless. It's because of what a diff looks like.

A tag bump is a one-line change. In the diff, the changed line is right there and it reads as complete. The unchanged SHA on the next line isn't highlighted, isn't flagged, isn't in red or green. Its absence from the diff is invisible, and you can't review what the diff doesn't show you.

And the 'unknown' fallback survived for a similar reason. A fallback reads as defensive, like care was taken. Nobody reviews a graceful default as a bug; it looks like the opposite of one.

The systemic cause, then: two facts that must always agree were stored as two independent values, with nothing enforcing the agreement, and a failure mode that produced plausible output instead of an error.

How to avoid it

If a value can be derived, derive it. Don't store a SHA next to a tag and trust people to keep them in step; resolve one from the other, or record only one and compute the rest.

Where you genuinely must store both, add a check that compares them and fails loudly on a mismatch. It costs almost nothing and it turns a silent lie into a red build.

And be suspicious of defaults in provenance code. A placeholder like 'unknown' feels safe, but a provenance record's whole job is to be accurate, so an honest refusal beats a convenient guess every time.

Right, so, the two lines that disagreed. That's all it took.

Every commit, file reference and date above is checked against this incident's Noctem evidence pack before publication. Evidence: decksmade-vendored-charts-pin-named-wrong-commit.

Also in the museum

The other 23 failures