Notes from the lab bench.
Deep dives on model internals, training experiments, and engineering discipline — the work behind the systems we ship. Written for practitioners.
Why Couldn’t an LLM Pick Its Best Expert?
A hindsight oracle found better expert choices. A trained judge could not use them, and a prospective observability test found no reliable live signal. The failure points to a cheaper test every controller should pass before scaling.
I forced a Mixture of Experts to specialize by topic. It got worse.
The folk model says a Mixture of Experts holds a code expert and a medicine expert. Trained from scratch, the routing tells you more about whether a word is a comma than about what the document is about — and it froze in the first 2% of training. Three attempts to force topic routing all came out slightly worse.
What does a Mixture-of-Experts router actually read?
The folk model says MoE experts are specialists: a code expert, a geography expert. I aimed a Jacobian lens at the router on OLMoE and Qwen-MoE and read what actually flows into each expert. It sorts brackets, newlines, and verb tense, not topics.
The sparse model that lights up 93% of itself
OLMoE spends about 1B of its 7B parameters on any given token. I put CUDA events around every MoE layer on a 4090 and recorded which of its 1,024 expert slots really fire. At a batch of eight, 628 of them fire every step.
Before a model answers, it thinks in concepts
The prompt never says Italy, but twelve layers into Qwen3.5-4B the model is already thinking it. J-Scope lets you watch that happen, inject France and get Paris, or drag one vector and watch the answer walk from euro to yen. Live tool and a 72-second tour inside.
I fed a model facts through its J-space instead of its prompt
I wrote a China vector into a model's residual stream and it answered "Beijing" to a question that never mentions China. Then the cache turned out to beat the vectors. Full experiments, raw data, and where each method breaks.
Can you tell a good AI agent plan from a bad one before you run it? I spent 6 million tokens finding out.
Can a cheap reward model tell a good AI agent plan from a bad one before you run it? I reproduced Orch-RM on a consumer RTX 4090 and RTX 3090: the verifier barely beats a coin flip on the signal that matters, yet still beats majority vote. Cheap to run, expensive to teach.
I Gave My Coding Agent Karpathy's Discipline Rules. It Got Too Careful to Fix the Bug.
I tested the viral Karpathy coding-discipline rules on 100 real SWE-bench bugs, three times. They didn't help: the agent fixed fewer of them, trading fixes for caution.
I bolted MiniMax's MSA sparse attention onto a 3B model on a single 4090
MSA sparse attention, retrofitted onto a 3B model on a single RTX 4090 with no training: the 28.4x compute cut holds, quality survives, plus caveats the paper skips.
I ran DFlash on a MacBook. The 4x is a data-center number.
I tested DFlash speculative decoding on an M5 Max and RTX 4090. The 4x headline is real, but workload and concurrency decide whether you get a speedup or a slowdown.
Why AI Still Struggles with Negation and How We Can Fix It
LLMs flounder at one of the most basic cognitive tasks: understanding negation. Not because the task is too complex, but because their design wasn’t built for it — and in medicine, law, and finance, a misread “not” is dangerous.
For client stories and business outcomes, read Insights →