The crash that only hit players who were good at my game

The crash reports for Overbloom had a pattern I did not want to believe. They clustered around high scores. Not a specific obstacle, not a specific device model in isolation, but long runs. The better someone was at the game, the more likely the app was to die on them.

That is close to the worst possible failure mode. The players it hits are the ones most invested, and they hit it at the exact moment they are about to beat their best score. It also survives every test you would normally run, because nobody QAs a game by playing one run for twenty minutes.

Why nothing caught it#

My test loop was a run or two at a time. Launch, play, die, check the thing I changed, repeat. Even the automated smoke scripts drove short sessions. Every one of them passed with plenty of headroom.

The leak needed roughly twenty minutes of uninterrupted play before it crossed the line, and it only crossed the line on devices with 2 to 3GB of RAM. My daily driver had more memory than that, so on my own phone the leak was real and invisible. It simply never got large enough to matter before I stopped playing.

Splitting the problem in half#

The engine is plain TypeScript operating on a plain object. No Skia, no React Native, no native modules in the simulation itself. That made the first diagnostic step cheap: run the sim headless and see whether the leak is in my code or underneath it.

/**
 * Headless memory soak for the game sim (pure math, no Skia, no RN). Run with:
 *
 *   bun --expose-gc scripts/mem-soak.ts [minutes]
 *
 * Drives update() through one continuous god-mode endless run (no deaths, the
 * 100k+ score profile players crash at) and samples, once per simulated
 * minute: forced-GC heap size, dynamic-array lengths and high-water marks,
 * and per-minute fx event rates.
 */
 
const s = makeInitialState(WORLD_H);
resetState(s, WORLD_H);
s.mode = 'play';
s.dbgGod = true; // survive forever: the 100k+ profile is one uninterrupted run
 
const ptr: Pointer = { x: W / 2, y: WORLD_H * 0.78, holding: false };

The reasoning is written into the file header, and it is the part I would transplant to any other project:

Retained heap growing linearly with simulated time = a sim-side leak; flat heap shifts suspicion to the native layers (Skia/audio/ads) the sim never touches.

Forty-five simulated minutes ran in seconds and the heap was flat. The sim was clean. That single result eliminated the two thousand lines I would have started reading first, and pointed at the native layers instead.

The script also samples something less obvious that turned out to matter. It counts how many effect events fire per simulated minute, because each one is a sound effect or a haptic on the JS thread in production. If you want to know what native churn looks like late in a run, that number tells you before you ever pick up a device.

The actual bug#

Every sound effect leaked. Permanently.

The audio engine’s native event registry holds the onEnded callback as a C++ garbage collection root, and the callback captured the source node. So the native side kept the closure alive, the closure kept the node alive, and neither side would ever release the other. On top of that, setBuffer makes the node its own deep copy of the decoded clip.

Late in a run the game fires roughly five sound effects a second. Each one pinned a node and a copy of a decoded audio buffer forever. That worked out to somewhere between 50 and 80MB per minute, which is exactly the shape of a crash that arrives around the twenty minute mark on a 3GB phone.

The fix is to unpin both halves by hand at the moment the voice finishes:

// Release the voice DETERMINISTICALLY, not at some future GC. The native
// registry captures the node, so without `onEnded = null` neither side can
// ever free the voice.
voice.onEnded = null;
voice.buffer = null;
voice.disconnect();

There was an error path to handle too. If the engine throws mid-play, onEnded may never fire for that voice, which means the cleanup never runs and the voice counts against the live-voice cap forever. Enough of those and sound goes silent for the rest of the session. A half-built node has to be torn down in the catch block for the same reason, or its registry pin and buffer copy leak just the same.

The other three#

Once I started looking at native memory rather than JS memory, the audio leak turned out to have company.

Music was decoded far too eagerly. Both gameplay tracks were decoded at boot, the equipped theme’s overrides stacked on top while keeping the base as a fallback, and two screen tracks stayed warm. At full stack that was around 400MB of float32 PCM resident, which on a 3GB device means the system starts sending memory warnings and then kills you. Making gameplay music decode lazily with exactly one resident buffer brought steady state down to roughly 130 to 150MB. Crossfades still sound identical, because a live layer holds its own reference to its buffer across the swap.

Skia objects were waiting on Hermes finalizers. The game records a fresh Picture every frame, and a black hole spawns 60 to 120 ImageFilter objects a second while it is on screen. Each of those is a small JS object holding a large native allocation, so the JS heap looks fine while native memory climbs. Hermes has no idea it should collect urgently. They now get disposed on an explicit delay instead:

// Dispose per-frame Skia natives deterministically instead of waiting on a
// Hermes GC finalizer, the lag that ballooned native memory and jetsammed
// low-RAM devices mid-run.
const pictureAging = useSharedValue<SkPicture | null>(null);
const pictureRetired = useSharedValue<SkPicture | null>(null);

Two frames of delay, because the compositor may still be reading last frame’s picture when this frame’s is recorded.

Ads were preloading too early. A warm rewarded video creative is tens of megabytes of resident native memory, and it was being loaded at boot and again right after each ad closed, so it sat there for the entire run. Moving the load to the death transition means it arrives during the results sequence, which is the only window where one is about to be shown anyway.

What I took from it#

Native memory and JS memory are separate problems and most tooling shows you the wrong one. My JS heap was healthy through every one of these bugs. The number that mattered was resident native memory, and nothing in my normal workflow put that in front of me.

An engine that runs headless is worth building for this reason alone. Being able to answer “is it my code or the platform” in ten seconds changed a week-long hunt into a targeted one.

Deterministic cleanup beats a finalizer whenever a small JS object owns a large native allocation. Skia pictures, audio buffers and image filters all fall into that category, and none of them are things Hermes can reason about.

And the general lesson, which I keep relearning: soak testing finds an entire class of bug that no amount of feature testing will. If your app has a session that can run long, something in it probably leaks.

Overbloom is on the App Store, and long runs survive now.