Why Couldn’t an LLM Pick Its Best Expert?

Something should have happened. It didn’t. Why?

We built a judge for an LLM.

The normal system chose its experts before seeing their work. Our judge could wait, inspect the completed expert outputs, and then change the mixture. It should have made a better choice.

Instead, after almost 2.7 billion training tokens, the model with the judge was worse.

To understand why, we ran three experiments. First, we let the future choose the best expert. Then we trained a real judge. Finally, we tested whether the useful choice could be predicted from information available at the time.

First, what is an LLM expert?

The word expert is misleading. An LLM expert is not a tiny lawyer, programmer, or physicist inside the model. In a Mixture-of-Experts model, an expert is simply a learned computational pathway: a small neural network inside certain transformer blocks.

1. The model turns the current token and its context into a long list of numbers called a hidden state.
2. A router scores that hidden state and selects a few expert pathways.
3. Each selected expert transforms the same hidden state.
4. The model mixes their outputs and continues processing.

If two experts produce outputs e₁ and e₂, a simplified mixture might be m = 0.62e₁ + 0.38e₂. That result is not the final answer. It becomes another internal state passed to later attention and expert blocks.

Experts may specialize in shapes found in hidden states, but that does not make them human-style subject specialists. The token “bank” has a different state in “deposit money at the bank” than in “sit by the river bank.” Different pathways can respond to those shapes without one being a finance expert and the other a river expert.

The real question was narrower: after several experts finish their calculations, can a judge identify which output will lead to a better continuation before that continuation happens?

Experiment 1: let the future choose

We began with OLMoE, a pretrained model with 64 experts in each MoE block. Its router activates eight at a time. At one position, we tried those eight experts separately, looked at the next 32 tokens, and measured which choice would have produced the lowest negative log-likelihood, or NLL. Lower is better.

This was deliberately a hindsight test. A live model cannot see the correct future before choosing. We used the future only to ask whether better choices existed.

The router was good at choosing a useful set. Its ranking did not say which single expert would look best later. Forcing only its top-ranked expert made NLL 0.00601 worse.

The catch: hindsight gets free luck

The oracle appeared to improve NLL by 0.01595. But imagine rolling eight dice and reporting only the lowest roll. It looks unusually good even when the dice contain no useful information.

We repeated the procedure with nonsense candidates: expert outputs borrowed from unrelated text positions. They should not help the current token, but the oracle still got eight attempts and kept the luckiest. Those unrelated outputs appeared to improve NLL by 0.01219.

This trap extends beyond experts. If an experiment chooses the best prompt, layer, checkpoint, or intervention using the same outcome it later reports, part of the gain comes from search. A fair control needs the same number of choices and the same selection process.

Still, the corrected effect was not zero. With perfect hindsight across all 16 blocks, a cleaner control improved NLL by about 3.6%. A better path existed. The question was whether a model could find it without seeing the future.

Experiment 2: train the judge from the beginning

Perhaps adding a judge to a pretrained model was too late. We trained a smaller MoE model from scratch with an output-aware judge built in. Four selected experts finished their calculations, then a small attention module could change their weights.

• Both models ran the same experts.
• The judge changed weights but could not add a new transformed output.
• Both models began with the same shared weights.
• At step zero, the judge exactly reproduced the normal mixture.
• The judge added only 0.316% more parameters.

The judge was active. Its weights moved away from the router and became more decisive. But after 2.697 billion tokens, its NLL was 0.00285 higher than the control’s, and the gap widened over training.

The result suggests a better mental model: experts are ingredients, not contestants. One expert output can be useful only in combination with the others and with later layers. Our judge treated an orchestra like a talent show.

Experiment 3: could the present predict what the future knew?

The oracle’s label came from future tokens. A deployed judge cannot see those. We tested the information boundary directly on 400 sequences across all 16 MoE blocks.

Prediction models received only information available at decision time: the hidden state, router scores, selected expert identities, completed local expert outputs, and differences between those outputs. We excluded future tokens, future losses, gradients, and downstream counterfactual states. We split by sequence so the same text could not leak between train and test.

At the final block, hindsight offered 0.014187 NLL of gain. A calibrated ridge policy captured none of it; its held-out intervention was 0.000442 worse. A bounded XGBoost check with router, hidden-state, and expert-output features chose the same policy every time: never intervene.

More parameters can learn a more complicated pattern. They cannot create predictive signal absent from their inputs.

The answer in plain language

“Best” was defined with information that arrived later. The router selected a useful group using the current state. The experts produced pieces of a shared computation. Hindsight replayed alternatives against the correct future and named a winner. The decision-time signals we tested did not reliably reveal that winner in advance.

1. An LLM expert is a pathway, not a person with a subject specialty.
2. Router confidence does not measure future usefulness.
3. An oracle can look powerful simply because it gets many chances.
4. Expert outputs may work together like ingredients.
5. A better action can exist without being predictable in time to use it.

The cheaper experiment to run first

We found a large hindsight opportunity, trained a new architecture for billions of tokens, and only then asked whether the opportunity was predictable. The last two steps should have been reversed.

1. Define exactly what the live system can see.
2. Measure the hindsight ceiling.
3. Subtract free search with a matched control.
4. Test whether a cheap policy recovers gain on untouched data.
5. Only then train the expensive architecture.

Before scaling this idea, can information available now predict the action hindsight prefers later?

What we are not claiming

This does not prove that output-aware expert selection can never work. The prospective test used one pretrained MoE family, 400 sequences, one intervention style, and a fixed set of decision-time features. Another model, objective, or controller with memory may behave differently.

We also did not complete one planned control: a simpler output-aware chooser without cross-expert attention. The training result applies most directly to the attention-plus-entmax design, not every output-conditioned router.

The narrower result is that the normal weighted mixture beat hard top-one routing, a trained output-aware judge, a linear policy, and a bounded nonlinear check. A large hindsight opportunity did not become a useful live policy.

Something should have happened. It didn’t. Now we have a better answer to why.

This is an exploratory internal experiment, not a peer-reviewed result. The implementation and durable artifact history live in the Phenx research workspace. Related context includes work on hidden-state geometry and expert specialization, expert-router coupling, and causal expert control.