Field notes from the shop floor.
Short, practical pieces on industrial AI, process control, and the margin hiding in data you already collect.
I forced a Mixture of Experts to specialize by topic. It got worse.
The folk model says a Mixture of Experts holds a code expert and a medicine expert. Trained from scratch, the routing tells you more about whether a word is a comma than about what the document is about — and it froze in the first 2% of training. Three attempts to force topic routing all came out slightly worse.
What does a Mixture-of-Experts router actually read?
The folk model says MoE experts are specialists: a code expert, a geography expert. I aimed a Jacobian lens at the router on OLMoE and Qwen-MoE and read what actually flows into each expert. It sorts brackets, newlines, and verb tense, not topics.
The sparse model that lights up 93% of itself
OLMoE spends about 1B of its 7B parameters on any given token. I put CUDA events around every MoE layer on a 4090 and recorded which of its 1,024 expert slots really fire. At a batch of eight, 628 of them fire every step.
Before a model answers, it thinks in concepts
The prompt never says Italy, but twelve layers into Qwen3.5-4B the model is already thinking it. J-Scope lets you watch that happen, inject France and get Paris, or drag one vector and watch the answer walk from euro to yen. Live tool and a 72-second tour inside.
I fed a model facts through its J-space instead of its prompt
I wrote a China vector into a model's residual stream and it answered "Beijing" to a question that never mentions China. Then the cache turned out to beat the vectors. Full experiments, raw data, and where each method breaks.
Can you tell a good AI agent plan from a bad one before you run it? I spent 6 million tokens finding out.
Can a cheap reward model tell a good AI agent plan from a bad one before you run it? I reproduced Orch-RM on a consumer RTX 4090 and RTX 3090: the verifier barely beats a coin flip on the signal that matters, yet still beats majority vote. Cheap to run, expensive to teach.
I Gave My Coding Agent Karpathy's Discipline Rules. It Got Too Careful to Fix the Bug.
I tested the viral Karpathy coding-discipline rules on 100 real SWE-bench bugs, three times. They didn't help: the agent fixed fewer of them, trading fixes for caution.
I bolted MiniMax's MSA sparse attention onto a 3B model on a single 4090
MSA sparse attention, retrofitted onto a 3B model on a single RTX 4090 with no training: the 28.4x compute cut holds, quality survives, plus caveats the paper skips.
I ran DFlash on a MacBook. The 4x is a data-center number.
I tested DFlash speculative decoding on an M5 Max and RTX 4090. The 4x headline is real, but workload and concurrency decide whether you get a speedup or a slowdown.
Why your adaptive controller might be causing the defects it’s trying to prevent
When cutting force climbs, the controller slows the tool. But on many processes the damage is in the dwell, not the peak, so the “safe” move quietly wears the part. Here is how to find it in your own data.