Acquisition — record your own
You are looking at a recording that ships with this repo. To point the instrument at your own prompts, clone it and start the local runtime — the model runs entirely on your machine:
git clone https://github.com/phenx-inc/moe-lightup
cd moe-lightup
pip install torch transformers
python serve.py # → http://localhost:8000
Works on CUDA, Apple Silicon (MPS) or CPU. No GPU is required — CPU is just slower, so drop the decode-step count. Nothing leaves your machine.
Expert activation map
A dense network pushes every token through every parameter. This model replaces each feed-forward block with 64 independent experts plus a small router that scores them and keeps the best 8. The remaining 56 do no work for that token. That is how 7B stored parameters cost roughly 1B of compute per token.
Each row below is a layer, each column an expert. The band sweeping downward is real: layer 1 cannot start until layer 0 finishes, because layer 0's output is layer 1's input. Isolate a domain to see what one subject alone touches.
Layer routing — one token's path
MoE is often described as if a token gets classified once — "this is code, send it to the code expert." The trace says otherwise. The router runs independently in all 16 layers and re-picks 8 of 64 each time, so a single token makes 128 expert visits on its way through the stack.
Read the carry-over figure below: on average only ~1 of 8 experts survives from one layer to the next. There is no stable "code expert" — the selection is rebuilt from scratch at every depth.
The bar behind each expert is its gate weight — the router's confidence. They are strikingly flat: the top pick usually carries only 7–11%, not 90%. The model is blending eight opinions rather than consulting one specialist, which is why swapping a low-ranked expert is often survivable — and why it is never free.
Batch loading — where serving cost is decided
Pulling an expert's weights out of memory costs the same whether 1 token needs it or 50. So inference engines pack many sequences into one step: the experts they collectively want get paid for once, not once per user. Below, the same trace at batch 8 and batch 32.
Four times the tokens for a quarter more time. 8 → 32 sequences lights 61% → 93% of all expert slots while the step grows only 30.3 → 38.1 ms. The arithmetic was never the bottleneck — hauling expert weights out of memory was, and a batch amortises that haul across everyone in it. It is also why a GPU that looks busy and one that looks idle can cost nearly the same.