Interpretability & inference·July 19, 2026·6 min read

The sparse model that lights up 93% of itself

A Mixture-of-Experts is supposed to use a fraction of itself per token. I recorded which experts actually fire, and the fraction is not what it looks like.

A 16 by 64 grid of expert slots. Layer 13 is executing, outlined in amber; the twelve layers above it are already lit from earlier in the same decode step. — One decode step, batch of eight. Layer 13 is executing; every dimmer cell above it already fired during this same step.

A Mixture-of-Experts model is the efficient one. OLMoE holds 7 billion parameters and spends about 1 billion on any given token, because each of its 16 layers carries 64 separate experts and a small router that keeps only the best 8. Seven-eighths of the model sits out every token. That is the whole pitch, and it is true.

I wanted to watch it happen, so I put CUDA events around every MoE layer of OLMoE-1B-7B on an RTX 4090 and recorded which experts fired at every step of a real decode. Sixteen layers times 64 experts is 1,024 expert slots. I expected a mostly dark grid with a thin trail of light moving down it.

With eight sequences decoding together, an average of 628 of those 1,024 slots fired on every step. Sixty-one percent. I raised it to 32 sequences, still a small batch by serving standards, and it went to 93%. Across the entire run, three slots never fired at all.

The sparsity is real. It is just per-token, and nothing in production serves one token at a time.

The whole argument in 85 seconds. Every counter animates to a measured value from the recording.

The router runs sixteen times, not once

The tidy mental model of MoE is that a token gets sorted. This is a code token, so it goes to the code experts; that one is math, so it goes elsewhere. The recording says otherwise, and the correction is more interesting than the original picture.

Routing is not a decision the model makes about a token. It is a decision each layer makes, independently, about the vector it is currently holding. Sixteen layers, sixteen routers, sixteen fresh picks of 8 from 64. A single token makes 128 expert visits on its way through the stack.

Follow one token in the trace and the experts barely repeat. At layer 0 it takes experts 59, 8, 39, 43, 6, 31, 32 and 29. By layer 4 it is on 26, 50, 58, 57, 49, 36, 1 and 60, with nothing carried over. Across all 3,072 token-layer pairs I recorded, the average number of a token's eight experts that survive from the previous layer is exactly one. A single token ends up reaching 56 of the model's 64 experts by itself, and the thriftiest one in my batch still touched 51.

Sixteen columns, one per layer, each listing that layer's eight chosen experts with their gate weights. Almost no expert appears in two adjacent columns. — One token's sixteen routing decisions, left to right. Each column is a layer's top-8; almost nothing carries across.

The gate weights say something similar. If routing were classification you would expect the winner to dominate, and it doesn't. The median first-place expert carries 8.6% of the weight, the middle half land between 7% and 11%, and 93% of them sit below 15%. The model is averaging eight modest opinions rather than consulting a specialist. That is worth knowing before you assume any particular expert is load-bearing, and it is why swapping one for another is sometimes survivable, which is a different article.

What batching actually buys

Each expert in this model is two matrices, 12.6 MB of them in bfloat16. All 64 of them, in all 16 layers, live in the GPU's memory the entire time; 12.9 GB, resident, never moving.

Resident is not the same as usable. Tensor cores cannot multiply weights that are sitting in HBM. The weights have to be streamed into the tiny pool of on-chip memory next to the cores before any arithmetic happens, and that pool is thousands of times too small to hold them, so they get streamed again the next time the expert runs. The hauling is the cost. The arithmetic, for a handful of tokens, is nearly free.

This is why batching works, and why it works so lopsidedly. Pull an expert's 12.6 MB in once and every token in the batch that wanted that expert rides along on the same haul. Going from 8 sequences to 32 in my recording cost 30.3 ms per step to 38.1: four times the tokens for a quarter more time. Per token it fell from 3.79 ms to 1.19, and throughput went from 264 tokens per second to 839.

Two expert-activation grids side by side. At batch 8 about 61 percent of cells are lit; at batch 32 about 93 percent are lit. Bars compare 30.3 ms and 38.1 ms per step. — Four times the tokens lights 61% to 93% of the grid and costs 26% more time per step.

That is the same trade that lights up the grid. More sequences means more distinct experts requested per layer, 39 of 64 at batch 8 and 59 at batch 32, and the model gets wider and cheaper per token at the same time. A GPU serving one user and a GPU serving thirty-two are touching almost the same weights, and one of them is doing thirty-two times the work.

Play the recording yourself

Everything above came out of one trace, and the trace is embedded below rather than described. Press RUN and the wavefront sweeps down the layers in the time it actually took; the timeline under the grid is real microseconds. Pick a domain chip to isolate a single subject and the grid thins out to exactly 128 cells, which is one token's 16 layers times its top-8.

moe·lightup — the recorded trace, playableOpen full-screen ↗

The real recording, playable. Hit RUN in panel 01 and watch the wavefront sweep down the 16 layers in measured time, or isolate a single domain to see what one subject alone touches. Panel 00 is the recorder; it turns into a prompt box when you run the tool locally. Best on a desktop screen.

The measurement I nearly got wrong

To time individual experts you have to find where they execute. In current transformers, OlmoeExperts.forward contains a readable Python loop over the experts that were hit, one at a time. I copied it, added a CUDA event on each side of every expert's matmuls, and had per-expert timings within an hour.

Then I checked the copy against the original and the logits disagreed. Two percent of argmax positions came out different, which is far too much for reordered floating point.

The loop I had instrumented was not what the model runs. from_pretrained quietly sets grouped_mm, and a layer's experts go through one fused grouped GEMM. The Python loop is the fallback. Both are real code paths, and the difference is not academic: same model, same routing, 30.3 ms per step fused against 162.7 ms in the loop, and roughly ten times that ratio inside the MoE layers themselves.

My instrument was five times slower than the thing it was measuring, and it was slower in exactly the way that would flatter my hypothesis. I had spent a week on the theory that touching fewer distinct experts saves real time, and the loop bills you per expert, so the loop agreed with me enthusiastically. The fused kernel is much less impressed.

Once configured honestly, my instrumented forward matched stock eager bitwise, and the numbers above are all from the fused path that actually runs. The eager path stays in the recorder behind a flag, because watching experts fire one at a time is genuinely clearer than watching a layer flash at once. It is a diagram, not a measurement.

What this is, and what it isn't

One 7B model, one consumer GPU, batches of 8 and 32, greedy decoding of eight passages of prose, code, math, JSON and dialogue. Production MoE serving runs far larger experts across many devices with fused kernels and continuous batching, and at that scale the balance of these costs moves. A model with big enough experts becomes bound by tokens rather than by calls, and skew across experts starts costing real idle time in a way it never did here.

What I would carry across scales is narrower. Sparsity is a property of a token's path, not of a served workload. A model that activates one-eighth of itself per token will still touch nearly all of itself once you batch enough users to be worth deploying, and the memory system will be doing roughly the same work either way. When someone says a Mixture-of-Experts model only uses a fraction of its parameters, ask which fraction, and over what.

The instrument is open source. It records your own prompts on your own machine, CUDA or Apple Silicon or CPU, and plays the result back as the grid above: github.com/phenx-inc/moe-lightup. Type in something the model has never seen and watch how much of it wakes up.