Killing every `bun` process by name to stop a hung test took down 13 services that systemd would not restart

Restart=on-failure
clean SIGTERM

# The Day a Hung Test Took Out Every Discord Plugin

On 2026-10-15 a test-running subagent decided a browser test had hung, and it went looking for something to kill. What it killed was thirteen systemd units and the Discord plugin inside all eight interactive sessions. Nothing crashed. That's the part I keep coming back to.

Symptom

The first sign was silence. Sessions that normally answered on Discord just stopped answering, and nothing anywhere said anything was wrong.

The numbers, since this site cares about numbers: 13 units killed, 8 interactive sessions affected, about 20 minutes of outage. Every health check stayed green throughout. The claude process in each session was alive and well, so the checks were happy. The thing people actually talk to had quietly gone missing from inside it.

And the unit state told the same boring story. The units were configured with Restart=on-failure, and they were not restarted. No failure was logged, because as far as systemd was concerned there hadn't been one. The units had exited cleanly, a clean SIGTERM, exit code 0, nothing to see.

Twenty minutes is not long. But it was twenty minutes of nobody knowing, which is a different thing from twenty minutes of a known outage.

What we thought it was

My first guess was the plugin itself. Discord integrations drop connections all the time, so a plugin quietly falling over seemed absolutely believable. Tokens expire, gateways reconnect badly, a websocket goes stale and never recovers. I've seen all of these, you know, and each one looks exactly like this from the outside: the bot is present in the process list and absent from the chat.

So I went down that road first. I looked for connection errors, for a crash inside the plugin, for anything that explained one plugin dying. The trouble was that it wasn't one plugin. It was all eight sessions at once, and thirteen units besides, which is not how a flaky network behaves. Flaky networks are messy and uneven. This was tidy. Everything affected had died together, and tidy is a clue.

The second wrong idea was a systemd restart policy that simply wasn't working. That was believable too, because a service that stays dead after dying looks exactly like a broken restart policy. But the policy was fine. It was doing precisely what I had told it to do, which, anyway, is the uncomfortable bit.

Root cause

The trick was small and almost embarrassing in how simple it was.

A subagent running tests hit a hung browser test. To clear it, it piped ps into xargs kill, aimed at every bun process on the machine. Not the test's bun process. Every bun process. And the fleet units, along with the Discord plugin inside each interactive session, were bun processes too.

So the kill landed on all of them. The signal was SIGTERM, the polite default that kill sends when you don't ask for anything else, and bun handled it the way a well-behaved process should: it shut down cleanly and exited 0.

Here's where Restart=on-failure matters. That setting tells systemd to restart a unit only when it ends badly: a non-zero exit code, an unhandled signal, a timeout. A clean SIGTERM that ends in exit code 0 doesn't match any of those. To systemd that looks like somebody asking the service to stop and the service obliging, so it leaves the unit stopped. Which is correct. It's the whole point of the setting. We just hadn't thought about what else could produce that same signature.

The health checks failed to catch it for a separate reason. They watched the claude process. The claude process was fine, because nothing had touched it. The plugin lived inside it as a child, and the plugin was the part that vanished, so the check was measuring the wrong layer and reporting health about something that had never been in danger.

The fix

The fix has two parts, and only one of them is about the original incident.

The direct one is a new cron check. It looks for each session's Discord plugin, and if the plugin has gone missing for two runs in a row, it restarts that session. Two runs, not one, so a plugin that is merely mid-restart doesn't get bounced for no reason. That's the part that closes the 20-minute gap: the check now watches the thing people actually depend on, rather than the process wrapped around it.

The other part is less tidy. The test-running subagent no longer gets to do a blanket kill by process name. Stopping a hung test means stopping that test's own process, identified by something that belongs to it, not by a name it shares with half the machine.

Why it happened

This one survived review because every individual piece looked sensible on its own.

The kill command is a thing people write without thinking, and in a throwaway shell it's even fine. The restart policy is the standard, recommended setting, and nobody reviews a standard setting. The health check watched a real process that really was running. Put each one in front of a reviewer separately and each passes, I mean it, each passes.

What nobody reviewed was the gap between them. A name-based kill, plus a policy that honours a clean exit, plus a check that looks one layer too high, adds up to a way to take down everything without a single alarm. Review looks at changes, and this wasn't a change. It was an interaction that had been sitting there waiting for a hung test to trigger it.

There was also a quiet assumption that nobody had written down: that a process dying means something will notice. That held right up until the exit looked healthy.

How to avoid it

If you run anything like this, a few things transfer cleanly.

First, never kill by process name when you can kill by identity. A name is shared. A process ID, a process group, or a unit you started yourself belongs to one thing only, and that's the property you want when you're about to send a signal.

Second, know what your restart policy treats as intentional. Restart=on-failure is not a promise that the service will always come back. It's a promise that it will come back if it fails, and a clean exit isn't a failure. If a stray SIGTERM can reach a unit, decide deliberately whether that should count.

Third, and this is the big one, make your health checks measure the thing that matters, not its container. A check that confirms the wrapper is alive tells you very little about what's inside the wrapper. Check the feature, the plugin, the actual behaviour someone would complain about.

Finally, give automated agents narrower hands than you'd give yourself. A subagent clearing a hung test shouldn't have a command that reaches past the test.

Twenty minutes, thirteen units, eight sessions, and one tidy clue that nobody read until later.