Our AI-recall test counted "mentioned" if the answer contained the phrase, but the question already contained the phrase

knew = the answer explains the marker

# The Canary That Sang for Everything

The probe told us every assistant knew about our made-up phrases. Every single one. Which is a lovely result, right up until you remember the phrases were made up.

I want to walk through how that happened, because the bug was small, the cause was dull, and the lesson travels well beyond us.

Symptom

The canary probe in the noctemlabs repository plants invented marker phrases, then asks each AI assistant about them. The idea is simple. If an assistant can explain a phrase that exists nowhere except on our site, it must have read our site. That's the signal. Scoring was a single field called mentioned, which was true when the answer contained the marker.

There were six probe questions. Five of the six contained the marker inside the question text itself.

So I ran the test answers through it. Six of them. And mentioned came back true for answers that were plainly useless. An assistant saying "I don't know what that term means" scored as having mentioned the marker, because to say it doesn't know, it had to repeat the term. The echo was enough.

That's the symptom, stated plainly: the probe reported knowledge where the answer contained none.

What we thought it was

My first guess was contamination. Maybe the markers weren't as unique as we'd hoped. Maybe some assistant had genuinely picked them up, or the phrases collided with something real, and the probe was telling the truth about an awkward world.

It was a believable story. Canary designs fail that way all the time: you invent a "unique" string and the internet turns out to have an opinion about it. And a probe that lights up on day one looks, superficially, like either great news or a leaky marker. I spent a while on the marker side, checking whether the phrases could plausibly exist elsewhere.

Anyway, that was the wrong door. The phrases were fine. The scoring was the problem, and I only saw it when I stopped reading the verdicts and started reading the actual answers one at a time. Several "mentioned" answers were refusals. A refusal that happens to quote you is not a citation.

Root cause

The cause is almost embarrassingly small. The mentioned check was a containment test: does the answer string contain the marker string? You can see the shape of it at tools/canary-probe.mjs:113@225b81b^, the version before the fix.

Now pair that with the question design, visible at tools/canary-probe.mjs:14@225b81b^. Five of six questions embed the marker. So the question hands the assistant the exact string the scorer is hunting for. Any answer that restates the question, even to say it can't help, passes. "I don't know" passes. "Could you clarify what X means?" passes. A confident hallucination passes too, naturally, but so does honest ignorance, which is the part that makes the metric worthless.

I mean, think about what that does to the numbers. The measurement can't distinguish an assistant that has absorbed your content from one that politely declined. Both score true. A baseline taken that way wouldn't be noisy, it'd be uninformative, and worse, it would look healthy.

The score wasn't measuring knowledge. It was measuring whether the answer was on topic.

The fix

Commit 225b81b replaces the containment check with a new score, knew. The definition, as it appears in the output: knew = the answer explains the marker. That sentence is the whole design goal, and the code at tools/canary-probe.mjs:17@225b81b encodes it.

Two conditions have to hold. First, the answer needs at least two definition words that do not appear in the question. That's the part that kills the echo: if the only words matching are the ones we supplied, nothing was explained. Second, there must be no don't-know hedge, unless the site is cited. The exception matters. An assistant might say it's unsure but then point at our pages, and that's a legitimate hit, so a cited site overrides the hedge.

Then I rescored the six test answers with knew. All six came out correctly false. Six of six. No answer in the set actually explained a marker, and now the scorer agreed with a human reading them.

That was the check I cared about. The old score said yes where the truth was no; the new one said no where the truth was no.

Why it happened

The tempting answer is "nobody looked closely enough", but that doesn't actually explain anything. The real cause is that the scorer and the question set were written as if they were independent, when they weren't. The question template decided whether the marker appeared in the prompt; the scorer decided whether the marker appeared in the answer. Neither piece was wrong alone. The bug lived in the coupling between them, and a reviewer reading either piece in isolation would see sensible code.

It also survived because nothing in the pipeline had produced a result yet. There was no baseline to look wrong. A metric that has never been run against a known-negative case has never been tested at all, it has only been written. The change introduced the score on 2026-10-21, I detected the problem the same day, and the fix landed the same day, the day the runner was merged and before any baseline run. That is the lucky part, and I'd rather name it than pretend the process caught it. What caught it was reading raw answers instead of trusting a boolean.

How to avoid it

If you build any probe that checks whether a system "knows" something, a few habits transfer.

Run it against a case where you know the right answer is no, before you run it against anything else. A canary that can't return false hasn't been shown to work. Six known negatives rescored to false is a small test, but it's the one that mattered here.

Check what your own input leaks into the output. If the prompt contains the string you're scoring for, containment tells you nothing. Score on content the question did not supply.

Treat refusals as a first-class category. "I don't know" is an answer, and it's the most common wrong answer your detector will meet, so write the hedge handling deliberately rather than letting it fall through to true.

And read the raw answers at least once. Booleans compress away exactly the thing you need to see.

The marker phrases were never the issue, you know. The score was just answering a different question than the one we asked.