I forced a Mixture of Experts to specialize by topic. It got worse.

Every explanation of Mixture of Experts I have read describes a committee of specialists. One expert handles code, another handles medicine, another handles legal text, and a router reads each word and sends it to the right desk. It is a good story. It makes the architecture sound like an office.

I trained five language models from scratch on two consumer GPUs to check it, and the story is wrong in a specific and interesting way. In my models, knowing which expert a word was sent to tells you noticeably more about whether that word was a comma than about whether the document was a Python file or a novel. Worse, that arrangement was fixed within the first 2% of training and never moved again, while the model itself got five times better at its job.

Then I spent about sixty GPU-hours trying to force the committee-of-specialists arrangement into existence. It can be forced. It does not help.

This is the follow-up to an earlier study where I read a pretrained router's decisions with a Jacobian lens and found the same preference for token form. That result left an obvious objection open: maybe the pretraining data caused it. Training from random initialization closes that door.

Where a Mixture of Experts puts its experts

First, a note on vocabulary. Models do not work in words but in tokens, which are word fragments; “unbelievable” might arrive as three of them. I will say “word” throughout because the distinction changes nothing in what follows.

A transformer is a stack of identical blocks. Mine had twelve. Each block does two things to every word, in order: x = x + Attention(x), then x = x + MLP(x).

Attention is the part that looks at other words. When the model processes “sat” in “the cat sat,” attention is what lets that position pull information from “cat.” The MLP is the opposite: it processes each word on its own, with no knowledge of any other word in the sentence. Roughly, attention moves information between words and the MLP does the thinking about each word individually.

Mixture of Experts replaces the MLP only. Attention is left completely alone, in every block. So instead of one MLP per block you get eight, plus a small router that picks two of them for each word. In my models six of the twelve blocks were built this way.

The reason it is the MLP and not attention is unglamorous. The MLP is where the parameters are. Each of mine held 4.72 million weights against attention's 1.57 million, so replicating the MLP eight times is what actually buys you capacity. Eight copies across six blocks came to 226.6 million of the model's 311.5 million weights. Each word only used two of the eight, so the model stored 311.5 million parameters and spent 141.6 million on any given word. That gap is the entire trick: the memory of a large model at the compute cost of a small one.

The router decides per word, not per document

This surprised me when I first read the code. The router does not see the document. It does not see the sentence. It sees one word's vector and nothing else.

In my training setup each step processed 8 documents of 1024 words each. That is 8,192 words, and the router made 8,192 independent decisions per block, six blocks deep. Inside a single Wikipedia article, “The” might go to experts 3 and 5 while the very next word goes to 1 and 7.

Routing whole documents instead would have broken training outright. Each step would have produced 8 routing decisions rather than 8,192, spread across 8 experts. Most experts would receive nothing on most steps, get no gradient, and die. Per-word routing is not a refinement, it is what makes the thing trainable at all. It also means a document gets to use the whole model: across 1024 words it touches all eight experts, where routing by document would reach exactly two.

Two numbers, and what they mean

To ask what experts specialize in, I needed a way to measure it.

The first number is perplexity, which is how surprised the model is by real text. Formally it is the exponential of the average loss, but the useful intuition is: how many equally-plausible options is the model choosing between at each word? My vocabulary had 49,152 possible tokens, so a model that knew nothing would score about 49,152. After 50 million words of training mine were at 72.7. At the end they were at 14.4, meaning the model had narrowed roughly 49,000 candidates down to about 14 real contenders per word. Lower is better.

The second is normalized mutual information, which answers: if I tell you X, how much do you learn about Y? Zero means nothing at all. One means X tells you Y exactly.

I measured it twice for every model. Domain NMI asks how much the expert choice tells you about which of five sources the document came from: GitHub, Wikipedia, arXiv, StackExchange, or books, all from SlimPajama. Form NMI asks how much it tells you about the surface shape of the word, sorted into five buckets: whitespace, punctuation, numbers, alphabetic words, and everything else. If the committee-of-specialists picture were right, domain NMI would be high and form NMI would be low.

It came out the other way, and then it stopped moving

Here is the vanilla Mixture of Experts model, measured every 50 million words across 2.7 billion.

Perplexity fell by 80%. Domain NMI moved by 0.002. Form NMI moved by 0.004, in the wrong direction. Across 54 measurements spanning the whole run, the ratio of form to domain never left the range 1.43 to 1.59.

Routing carries about 45% more information about what a word looks like than about what the document is about. That part matches what the Mixtral team reported and what later work on part-of-speech sensitivity found, so I was reproducing a known result. Doing it from random initialization does rule out one alternative explanation, though: nobody can argue the pattern was inherited from a particular pretraining corpus, because there was no pretraining.

The freeze is the part I have not seen stated anywhere. The arrangement settled almost immediately. By 50 million words, 2% of the run, the router had picked its partition and it held that partition while the model underneath became unrecognizably better. Expert specialization did not emerge slowly over training. It never started.

I have a guess about why, and it comes from a single line in my own model code: self.experts = nn.ModuleList(copy.deepcopy(base_mlp) for _ in range(n_experts)). All eight experts begin as exact copies of each other. Identical weights. At step one the only thing distinguishing them is the router's random initialization. So whichever words the untrained router happens to send to expert 3, expert 3 gets gradient on, becomes better at them, and the router keeps sending them. The partition is an accident that then makes itself true. Surface form is simply the most legible axis in the early representations, so that is the accident you get.

Sparsity is worth a lot, whatever the experts are sorting

Before trying to fix the routing, it is worth saying what the architecture bought. Three of the five models were a matched-compute set: a plain model, a dense model with a double-width feedforward layer, and the Mixture of Experts. The dense-2× model and the MoE do exactly the same amount of arithmetic per word. The MoE just has a bigger library to choose from.

The MoE finished at 14.4025 against the matched-compute dense model's 15.5365, a 7.30% win, using 2.2 times the stored parameters. That gap held between 5.35% and 7.46% across 45 matched measurements and grew, rather than closed, through the learning-rate decay. So the experts are doing real work. They are simply not doing the work the folk model claims.

Three ways to force the issue

If routing by topic is the good arrangement and gradient descent is missing it, the fix is to insist. I tried three ways of insisting, each an independent training run of 2.7 billion words against a control that shared its architecture, its random seed, its data, and its data ordering.

The first never showed the router a label. I added a term to the loss that penalized the router when the words of a single domain spread evenly across experts. Combined with the standard load-balancing term, which keeps overall expert usage even, this maximizes exactly the domain NMI quantity I was already measuring. The label affected only the loss, so at inference the model was an ordinary MoE.

The second and third handed the label to the router directly. This is a different proposition, because such a model needs to be told the domain at inference time too. There is a wrinkle worth knowing: a linear router cannot do much with a topic input. Whether you concatenate a domain embedding or add one to the hidden state, both reduce algebraically to gate(x) + f(domain), a per-domain nudge to the scores and nothing more. So I built the honest version of that ceiling, and also a version that escapes it by letting the domain scale and shift the vector the router reads.

Both of those started at exactly zero. Because torch.zeros draws no random numbers, I could add the new parameters without disturbing the initialization of anything else, and I checked it: every base weight was bit-identical to the control, and the untrained model produced a loss of 10.94829369 with and without the addition, to the last digit. The model had to learn to use the topic signal, starting from a state where it was ignoring it.

Forcing works. Pushing harder destroys it.

Before committing to a full run I swept the strength of the loss-based version on short 150-million-word probes, each against a control trained on the same short budget.

At strength 0.01 the router's domain information doubled and the form-to-domain ratio flipped from 1.43 to 0.70. Topic routing turned out to be easy to create.

The mechanism shows up in the form NMI column. At strength 0.01 it did not move, which means the model added topic structure on top of the surface-form structure it already had rather than trading one for the other. At 0.03 form NMI halved, and that is exactly where the perplexity cost appeared. The damage came from displacing surface-form routing, not from topic routing itself. Whatever the router was doing with word shape, it was load-bearing.

The last setting is the one I did not expect. Pushing ten times harder did not produce ten times the topic routing. It produced total collapse: the model abandoned most of its experts, load balance fell from 0.969 to 0.751, and domain NMI dropped to 0.0140, well below where it started. More pressure toward topic routing eventually destroys routing altogether.

Then the full 2.7-billion-word runs, and this is where the calibration misled me. At 150 million words strength 0.01 had looked free, at −0.03%. Over a full run it cost 0.87% on average and was behind the control at 35 of 35 measurements past the one-billion mark. The short probe was not wrong about its own conditions; it compresses the whole learning-rate schedule into 150 million words, which is a different regime from the one a long run spends most of its time in. Short probes are good for picking a setting and unreliable for predicting what that setting will cost.

I want to be careful about how much to claim from those small numbers, because I ran each configuration once. Two runs that should have been identical differed by 0.52% at one checkpoint, which is my only estimate of run-to-run noise. The 7.30% sparsity result is about fourteen times that and is safe. The 0.19% and 0.47% penalties are one to two times it, and the honest statement is that forcing topic routing produced no benefit, not that it cost a precise amount.

The one place it always failed

Breaking the final perplexity down by source explains the whole result better than the aggregate does.

The version that fed topic to the router beat the control on four of five sources and still lost overall, because books sit at perplexity 23.7 against everyone else's 3 to 8 and therefore dominate the average.

Neither intervention was broadly harmful. Both failed on books, and books are the source where the label carries the least information about the content. A GitHub file is reliably code. A book contains cooking, warfare, grief, dialogue, and weather. Telling the router “this is a book” and expecting it to route accordingly forces genuinely different words down the same path. The source label was a decent proxy for content in the homogeneous domains, a bad one in the varied domains, and the aggregate came out negative because the varied domains are the hard ones.

Why the router could not be talked into it

A paper from Johns Hopkins earlier this year gives the cleanest account of what I ran into. Because MoE routers are linear maps, similarity in the hidden state is both necessary and sufficient to explain similarity in expert usage. Specialization is a property of the representation the router reads, not something the routing mechanism creates.

That predicts the strangest number I collected. Feeding the domain label straight into the router produced less topic routing, at 0.230, than never showing it a label at all, at 0.454. Handing a linear router a fact does not reorganize how it partitions; the fact just becomes another small additive term. The loss-based version got further precisely because it applied pressure to the outcome and let the router find its own way there.

It also sets expectations correctly. Separate work this year found that replacing learned routing with a fixed hash costs only about 2 perplexity points against a learned router. If routing decisions barely matter compared to having the experts at all, nobody should expect better routing decisions to buy much. Which is roughly what the earlier work on domain-conditioned experts claimed, once I read it properly: DEMix built one expert per source domain back in 2022, and its reported wins were out-of-domain perplexity, modularity, and fast adaptation. In-domain quality was never the pitch.

There is also a practical corollary. If you were considering a topic classifier at inference time to drive routing, the oracle label is the ceiling any such classifier could reach, and the oracle label already loses. The predictor is moot.

The thing I would check next

There is a live possibility that I measured the wrong object, and I want to name it rather than let the null stand unchallenged.

My domain NMI number treats each block separately and averages the six. But a word makes six routing decisions on its way through the model, and 8 experts over 6 blocks gives 262,144 possible routes. Recent work argues that individual experts are ambiguous while whole routes are not, and that the meaningful unit in a MoE is the trajectory rather than any single choice. Knowing one digit of a six-digit code tells you very little. Knowing all six identifies it.

If that holds in my models, then a per-block measurement is structurally incapable of seeing the semantic structure, and every number in this piece is answering a narrower question than the one I asked. The measurement is cheap, since I still have the checkpoints. It is also easy to get wrong: with 262,144 possible routes and roughly 524,000 words in my evaluation set, most routes appear once or never, and mutual information estimated on a table that sparse produces impressive numbers out of pure coincidence. It needs a support threshold and a shuffled-label control, or the answer is worse than no answer.

The other cheap check is depth. Convolutional networks organize themselves from edges to textures to object parts, with the early layers coming out nearly identical no matter what you train them on. If transformers do anything similar, the early blocks would route on surface form and the late blocks on something more abstract, and averaging all six would report the mean of two opposite trends. I have the per-block tables already. I just threw the detail away.

We build and evaluate models on hardware you can actually buy, and we report the runs that did not work as carefully as the ones that did. If that is useful to you: info@phenx.io.