An elephant through a straw

The checkpoint is 137 GB. My GPU holds 24, the machine has 31 more, and everything that does not fit has to come off one NVMe drive at 6.2 GB/s. For one evening in the middle of this I thought I had it decoding at 12.9 tokens per second, which is inside the range I was aiming for and would have made this a much shorter article. What I actually had was a 284-billion-parameter model writing the word dekameters 108 times in 2,300 characters while my cache hit rate went to the moon.

Start with why any of this is possible. On any given token, most of a mixture-of-experts model does nothing at all. V4-Flash has 43 layers, each holding 256 expert MLPs plus one shared expert, and a router that picks 6 of the 256 per layer. Roughly 13B of the 284B parameters fire for a token and the other 271B sit there being expensive. So the working set per token is about 260 expert MLPs, or 3.4 GB at 4 bits, and not 137. At the drive's rated speed that is still 550 ms of reading per token, and the rating assumes a nice sequential stream rather than 260 scattered reads landing wherever the checkpoint happened to put them. Half a second per token, best case, before a single multiply. That is the hole to climb out of.

The checkpoint stays on NVMe in its shipped MXFP4 format and gets read with O_DIRECT, straight into pinned bounce buffers, going around the page cache entirely. This is not premature cleverness. Leave the page cache in the loop and Linux will happily evict 20 GB of scorching-hot expert weights to buffer a log file nobody will ever read. Then, because at this size a cache miss is the only thing that actually costs you, I re-quantized every single expert a second time down to 2.25 bits: 2-bit codes with one fp8 scale per group of 32 weights, threshold chosen by clip search. That took hours and produced 75 GB of additional files whose only purpose in life is to let the same RAM hold twice as many experts. Each FP4 expert is exactly 13,369,344 bytes, a number I can no longer unsee.

Not all six experts matter equally

The router says so itself. It emits six weights per token, and they are lopsided: usually one or two that carry the token and a tail that rounds to noise. If the model is going to tell you which of its own experts matter least, you may as well listen. So precision follows the router's ranking. The top a experts by weight always get FP4, fetched from disk if they are not cached. Ranks a+1 through b take FP4 when it happens to be resident and the 2-bit copy otherwise. Below b an expert is used if resident, dropped if not, and the survivors get renormalized to cover for it. That pair, (a,b), is the ladder. It is the only quality knob in the system and it is worth more than every caching trick I built.

Below is teacher-forced NLL against the FP4-exact model on held-out text, plotted against the speed each ladder actually decodes at on the slowest prompt in my set.

Hover or tap a point. The crossed-out point is a run I recorded as a win before reading its output. The next section has that output.

Ladder (2,4) costs 14.4% perplexity cold and reproduces the exact model's argmax on 92.6% of tokens, and warm pools mostly close even that gap. Then you take one step further, protect a single expert instead of two, and perplexity goes from 2.4 to 225. I had expected a slope. What is actually there is a trapdoor, and the far side of it is not degraded English, it is no English at all.

The evening the numbers lied

Which brings us back to dekameters. The run that logged 12.9 tok/s opened with a perfectly competent explanation of why LLM decoding is memory-bandwidth bound. Reading a model lucidly describe its own bottleneck while you are starving it of its own weights is a peculiar experience, and I recommend it. Four sentences in, it got to "LLMs generate text one" and then, instead of "token at a time," it produced dekameters. Then dekameters again, 108 times in total, increasingly interleaved with apl (149 of those), and at one point an end-of-repo-name special token, which has no business existing outside a code completion context and which I take as the model's way of filing a bug report.

Here is the part that makes this genuinely dangerous rather than merely funny. Looping is fast. A model stuck in a cycle asks for the same six experts every step, so every pool lookup hits, the drive goes quiet, and throughput climbs exactly as if I had done something clever. Speed and brain damage are positively correlated in this system. Every configuration that looked like a breakthrough was the model dying in a cache-friendly way, and I had been reading the throughput column instead of the text.

position .. apparent speed .. output ..

Verbatim output from the logged ladder-(1,4) run, truncated for length; it continues like that. Highlights are what the junk detector matches. The curve above is apparent tok/s, reconstructed from the run's logged 7.4 to 12.9 range and scaled by loop density (I logged aggregate rates only, not per-token ones). It rises as the loop tightens, because loops are cache-friendly.

So the harness grew a junk detector, which is a regex for the specific attractors this model falls into plus a repeated-trigram check. State gets snapshotted every 16 tokens, which is cheap because the KV windows and compressor states are small ring buffers. On a hit it rolls back to the last clean snapshot, forces every expert to full FP4 for 96 steps to shake the model out of the groove, then eases back down the ladder. Runs also stop at EOS now. They did not before, which means for a while I was counting several hundred tokens of post-EOS garbage as throughput, and yes, that inflated numbers I had already been pleased about. The damage is stochastic too: the same config gave me 1 artifact on one run and 192 on the next.

43 graphs, 43 syncs

With the drive under control the bottleneck moved to Python, at roughly 140 ms per token of launching thousands of tiny kernels. Each layer's dense work now goes into one captured CUDA graph: sliding-window attention, two compressed-attention paths, the hyper-connection mixing, the router, the shared expert. That mixing step deserves a mention because it contains a 20-iteration Sinkhorn normalization over a 4-stream lattice, which sounds exotic and cost 1.1 ms per call in stock PyTorch, for the crime of normalizing a 4x4 matrix. Fused into two Triton kernels it takes 69 µs, and there are dozens of those calls in every token. KV state lives in ring buffers indexed by position modulo window length, so replaying a graph never has to shift memory.

What a graph cannot swallow is the expert fetch, because which experts you need is decided by the router in the middle of the step. So each layer replays its graph, performs exactly one host sync to read the router's picks out of pinned memory, classifies them against the ladder, fires the grouped-GEMV expert kernels, and moves on. Python cost dropped from 140 ms to about 30.

It would have dropped there sooner if not for a line I am still slightly embarrassed by. Profiling said 339 ms per step were vanishing into nothing identifiable. The culprit was getattr(self, "pf_bounce", [t.pin_memory() for t in ...]), which allocates 140 MB of pinned host memory to build a default argument that is then thrown away, on every single call, because Python evaluates that third argument whether the attribute exists or not. It is a first-week Python fact and it cost me an afternoon.

At that point a token decomposed like this.

Where the ~260 expert lookups per token get served under each ladder, and what the token costs. Same machine, same disk; the only thing changing is how much quality you spend. Each row is a specific logged run; the (2,4) row combines tier counts from the Flask-tutorial run (130 ms) with the article run's 129 ms / 53 ms timing, which agree to within a millisecond. The crossed-out bar is the fake one.

Guessing the router two layers ahead

What remained was I/O nothing could anticipate: cold experts demanded in the middle of a token, about 53 ms of dead waiting. You cannot know layer 30's experts before layer 30's router runs. But the hidden state going into layer l's MoE already knows a great deal about what layers l+2 and l+3 are going to ask for, because that is what a residual stream is. So I trained a small MLP on it: hidden state plus a layer embedding in, 256 logits out per offset, supervised on 58k tokens of the model's own routing decisions. The training data costs nothing, because the model generates it by running. Its top-12 contains 75% of the true six experts two layers out.

Getting that dataset was less elegant. Capturing 58k tokens of routing meant 15 GB of tensors on disk, and my first training script loaded them by appending to a list and calling torch.cat, which momentarily needs twice the final size. On a 31 GB machine with the swap already full, the OOM killer ended it without ceremony. The loader now does two passes and preallocates.

Drag k. Recall = how many of the router's true top-6 appear in the predictor's top-k, measured on held-out tokens. The faint series is the same predictor trained on 15k tokens instead of 58k: 3.3× the data bought about four points, so this curve is close to its ceiling.

The predictions get prefetched from NVMe and promoted from RAM to GPU while earlier layers are still computing, and my first version of that made everything slower. Speculative copies were evicting genuinely hot experts to make room for guesses, so a wrong prediction cost me twice: once for the read, once for throwing away something I was about to need. The fix is a staging ring, a small round-robin region appended to the same slot buffers the expert kernels already index. A predicted expert lands there without evicting a resident one, and a wrong guess is simply overwritten by the next prediction. A right one is already in GPU memory when the router asks for it. That took the slowest prompt from 135.5 ms per token to 129.1 with zero artifacts, and in a deliberately cache-starved A/B it turned prefetching from a 6% loss into a 3% win.

Five things that did not work

Every one of these was built, run, and killed by a number. Together they cost more time than everything above.

Idea	Hope	Measured verdict
Speculative decoding with the shipped MTP head	2 tokens per fetch round	98% acceptance and still I/O-neutral, because the expert union across accepted tokens grows with the tokens, so the drive does the same work
Low-rank expert factorization	stream a rank-256 sketch instead of the expert	the experts are close to orthogonal; rank-256 leaves 80 to 92% of the energy behind
Activation-aware (AWQ-style) INT2	recover 2-bit quality with per-channel equalization	0.290 to 0.294 relative error, i.e. slightly worse; the clip-searched group scales had already taken that structure out
Reusing the previous token's experts as a prefetch set	36% of experts repeat token to token	they do, and it saves nothing, because a repeated expert is still resident from last step
Sync-free decode: drop every non-resident expert	zero host syncs, zero exposed I/O, one graph for the whole token	drops 26% of expert-uses, output becomes "the law of the law of the law of", and it was not even faster

That last row killed the design I most wanted to build: a single whole-token CUDA graph with a device-side expert table and no host synchronization anywhere in the step. It only works on a model that can absorb a missing expert, and this one cannot. The 2-bit tier is what stands between the ladder and "the law of the law of the law of," and keeping 2-bit copies resident for all 43 layers at once would need 78 GB. On a 24 GB card, the per-layer sync stays.

The number

7.75 tok/s at ladder (2,4), up from 1.7 with naive tiering under the stock HuggingFace forward, on audited transcripts with no artifacts. I was chasing 10 to 15 and I did not get there. The step is roughly 53 ms of exposed NVMe waiting plus 76 ms of everything else, and profiling says that "everything else" is mostly PCIe rather than arithmetic: at these cache sizes the GPU spends more time receiving experts than multiplying them. A perfect prefetcher puts the ceiling near 13 tok/s. Mine tops out around 80% recall and is flattening as I add data, so it funds maybe half the remaining distance. The rest would need either a second drive or a smaller model, and at that point you have stopped answering the question.

A late measurement, because it changed my mind about where the time goes. A CPU profile showed the decode loop spending 67 ms per token inside time.sleep, waiting on a 300 µs polling granularity in my own I/O completion loop. That looks like 67 ms per token of pure self-inflicted stupidity, so I replaced the poll with a blocking wait and ran five of each, forcing an identical token sequence so both variants demanded byte-identical expert traffic (38.1 NVMe fetches per token, every run). It is worth 2.7%: median 211.6 ms against 217.6. So the polling was never the problem. Nearly all of that sleeping is the drive being slow while my process has nothing to do, which is the same answer the GPU profiler gave from the other direction, and it is the most useful thing I learned about this machine. The slowest run of the ten hit 234.8 ms, for reasons that are the subject of the last paragraph.

One last confession, since it explains why the final numbers are worse than the ones in the middle of this article. Partway through the benchmarking my runs started failing with CUDA out-of-memory at pool sizes that had fit an hour earlier. I found two unfamiliar Python processes holding 6.6 GB of VRAM, assumed they were zombies from my own crashed jobs, and killed them both. They came straight back, because they were systemd units, because they were my own live product serving real traffic on the same machine. The 4090 that runs this experiment also runs a website. Everything after that point is measured on a box that is also serving a website, and labeled that way in the methods. It is a good argument against benchmarking on the same computer your users are sitting on.

Rig, model & methodology

GPU / RAM / disk	RTX 4090 24 GB · 31 GB RAM · 1× NVMe, ~6.2 GB/s sequential (O_DIRECT)
Model	DeepSeek-V4-Flash: 284B total / ~13B active, 43 layers, 256 routed experts (top-6) + 1 shared per layer, 3 hash-routed layers, MTP head; MXFP4 experts, FP8 dense
INT2 tier	2.25 bits/weight: 2-bit codes, per-32 fp8-e4m3 scales via clip search; 6.75 MB/expert, 75 GB total, ~0.3 per-expert relative output error
Caches (best config)	FP4: 1050 GPU slots + 1250 pinned-RAM slabs · INT2: 100 GPU + 500 RAM · LFU with frequency decay, seeded from a routing trace
Quality eval	teacher-forced NLL vs the FP4-exact model, 512 tokens of held-out technical prose; plus full-transcript artifact audits on every speed run, EOS-stopped
Speed	wall-clock over 384 to 1024 generated tokens, single stream, prompt 3 = a 1500-word article task, which is the slowest in the set because its cache locality is worst; per-layer CUDA graphs, greedy
Predictor	MLP 4096+128 → 1536 → 1536 → 2×256 heads (+2, +3 offsets), BCE on multi-hot top-6, trained on 58k tokens / 2.5M pairs of self-captured routing, 20k held-out
Numbers in the figures	every point is a specific logged run or eval file; the two "fake" points are preserved exactly as first recorded, because that's the point

Code is a local research stack: Triton kernels for MXFP4/INT2 grouped GEMV and FP8 dense, the tiered store, graph capture, repair machinery. Not yet published.

References

DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. arXiv:2412.19437. The architectural lineage V4-Flash descends from: fine-grained experts, shared experts, MTP.
A. Eliseev, D. Mazur. Fast Inference of Mixture-of-Experts Language Models with Offloading. 2023. arXiv:2312.17238. Prior art for MoE offloading with LRU caching and speculative expert loading on consumer hardware.
R. Hwang et al. Pre-gated MoE: An Algorithm-System Co-Design for Fast, Scalable Mixture-of-Expert Inference. 2023. arXiv:2308.12066. Deciding experts one layer early to hide the fetch; my predictor is the trained, two-layers-out cousin.
Open Compute Project. OCP Microscaling Formats (MX) Specification v1.0. 2023. opencompute.org. The MXFP4 format the checkpoint ships in.
NVIDIA. Getting Started with CUDA Graphs. developer.nvidia.com/blog/cuda-graphs. Same hero as last time.