One memory cap killed all four interactive sessions at once: they all shared the bot's cgroup

# Four sessions, one cap: a post-mortem of a cgroup OOM

On 2026-07-27 I lost four sessions at once, and the kernel did not even do most of the killing. That's the part worth writing down.

Symptom

The observed behaviour was simple and a bit rude. The three numbered work sessions and the bot's own session all died together. sessions_killed: 4. No gradual degradation, no warning, nothing in the sessions themselves that said "I am about to go". They were there, then they weren't.

It was detected the same day it happened, 2026-07-27, because the bridge went quiet and the numbered sessions went with it. Then it happened again. oom_rounds_same_day: 2. The second round was the one that told me this wasn't a one-off spike, it was a shape.

Two rounds in one day is a pattern, right? Anyway, that's what I took it as.

What we thought it was

The first diagnosis was memory pressure from the work itself. One of the sessions had grown too big and the kernel stepped in, so the others just got caught up in it when the machine ran short. That sounds reasonable because it is reasonable. Long-running sessions do bloat, and an OOM kill is exactly what you'd expect when one does.

It was believable for another reason too. The bridge service has a 2G ceiling (bridge_memory_max_gb: 2), and 2G is not roomy for several sessions plus whatever they spawn. So "too much stuff in too little memory" fitted the numbers. I mean, it fitted so well that I nearly stopped there.

But it didn't explain the part that bothered me. A single oversized process should lose one process. Why did four sessions go in one go, including ones that weren't the big one?

Root cause

Two mechanisms stacked, and neither is exotic. That's what makes it annoying.

The first is cgroup inheritance. Cgroup membership is inherited from the spawning process. So any session, tmux server or test run launched from inside the bridge lands in the bridge's cgroup, even if you give it its own tmux socket. A separate socket separates the tmux server from the default one; it does nothing about which cgroup the process lives in. I'd assumed the isolation was real. It was cosmetic. Everything I thought was a separate thing was quietly sharing the bridge's MemoryMax=2G.

The second is what systemd does after the kernel acts. When the cap was blown, the kernel killed exactly one process, which is what it's supposed to do. But the systemd default is OOMPolicy=stop, and that reads the OOM event as a reason to fail the whole unit. Then KillMode=control-group takes over and SIGTERMs everything left in the cgroup. One victim became four because the service manager escalated.

So the kernel did one small, correct thing, and systemd turned it into a mass termination. Neither piece is a bug, as such. They just compose badly.

The second round the same day fits this exactly. A vitest worker of about 936MB (vitest_worker_mb_approx: 936) was running inside the cgroup. That is a perfectly ordinary test worker, and it was enough to tip the shared 2G over again, because the headroom was being eaten by sessions I didn't know were counted against it. Same cap, same escalation, same result.

The fix

Three changes, all about making the blast radius match the thing that actually misbehaved.

The numbered sessions became systemd user units with OOMPolicy=continue and MemoryHigh=1536M. Moving them out of the bridge's cgroup means they no longer share its cap. OOMPolicy=continue means that if the kernel does kill one process, the unit carries on rather than being failed wholesale. MemoryHigh=1536M (bridge_memory_high_mb: 1536) adds a soft ceiling below the hard one, so a growing unit gets throttled and reclaimed before it ever reaches the point where the kernel has to kill anything.

The two system units got equivalent drop-ins, so the same protection applies there rather than only to the sessions I'd noticed.

Finally, sessions are now started only through a wrapper that never uses the default tmux socket. The wrapper is the thing that enforces the rule, so nobody has to remember it. That matters more than it sounds.

Why it happened

This survived review because every individual piece looked fine in isolation. A memory cap on a service is good practice. A separate tmux socket looks like isolation. The systemd default policy is a default, and defaults get trusted. Nobody reviewing any one of those would have flagged it.

The failure lived in the interaction, and interactions don't show up in a diff. The assumption that a separate socket meant a separate failure domain was never written down, so it was never checked. It was just a feeling that things were separate. And a feeling like that is very hard to review, because there's nothing on the page to disagree with.

How to avoid it

Treat the cgroup, not the process or the socket, as your unit of isolation. If two things must not die together, they need to be in different cgroups, and you should be able to prove it rather than believe it. Check where a process actually lives before you rely on a cap.

Read what happens after an OOM kill, not just whether one can occur. The kernel's decision and the service manager's reaction are separate, and the second is where a contained failure becomes a cascade. Decide on purpose whether a unit should continue or stop when one of its processes is killed.

Put a soft limit below your hard one. A high-water mark that throttles early is far kinder than a hard cap that kills.

And put your invariants in a wrapper, not in anyone's memory. If starting a session the wrong way is possible, someone will do it eventually. Make the wrong way unavailable.

Four sessions, one cap, and a default nobody had looked at. Anyway, that's the lot.