1
00:00:01,000 --> 00:00:07,873
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,873 --> 00:00:35,504
[Hal Turing] Today's paper is "PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference," by Yutong Lin, Yingdi Shan, Mingxing Zhang and colleagues at Tsinghua University, posted to arXiv in November 2026. The headline is a 1.56x decoding speedup on a single PC, with a giant mixture-of-experts model whose weights live on an SSD.

3
00:00:35,504 --> 00:00:51,340
[Dr. Ada Shannon] The question is whether giving each token a different number of bits per expert can coexist with a cache. Naively it can't, and that's the interesting part. I'd also say the 1.56x needs decomposing, because it bundles two different effects. We'll do that in a bit.

4
00:00:51,340 --> 00:01:19,483
[Hal Turing] Start with the setup. A mixture-of-experts layer has many expert networks, and a router picks a few per token, with weights that sum to one. Qwen3.5-397B-A17B picks 10 of 512 experts per layer. So only about 7.55 billion parameters are active per token, out of 386.55 billion routed-expert parameters. Sparse compute, but the storage isn't sparse.

5
00:01:19,483 --> 00:01:42,006
[Dr. Ada Shannon] Right, because the next token may pick any expert. So the local setup is this. The GPU runs attention, dense layers and routing. The CPU runs the selected experts. The full expert pool sits on an NVMe SSD, and host DRAM caches recently used ones. Batch size one, a single user. A cache miss means reading from the SSD while the token waits.

6
00:01:42,006 --> 00:01:43,817
[Hal Turing] And a good cache doesn't fix that?

7
00:01:43,817 --> 00:02:06,805
[Dr. Ada Shannon] Not enough. Their fixed-4-bit runtime has a 24 GiB cache, and it still reads 1.53 GiB from the SSD per token. That's about 87 milliseconds of exposed stall, 49.28% of decode latency, at 5.65 tokens per second. Nearly half of every token is spent waiting on storage. The exact-object hit rate there is 57.08%.

8
00:02:06,805 --> 00:02:11,496
[Hal Turing] So the obvious lever is fewer bits. And I assume it's not uniformly fewer.

9
00:02:11,496 --> 00:02:48,926
[Dr. Ada Shannon] Right, that's token-level dynamic precision. Different experts get different bit-widths on different tokens, under an average-bit budget. Two variations make that worthwhile. Routing weights vary by token, so a lightly weighted expert tolerates more error. And experts differ in how sensitive they are to quantization. The authors ran a headroom study on one layer, layer 30. Uniform allocation gives about 14.56% output error, and the best per-token choice from a restricted candidate set gives 10.59%. That's an estimate of potential, not a deployable result, and it's one layer.

10
00:02:48,926 --> 00:02:50,273
[Hal Turing] Okay, so where does it break?

11
00:02:50,273 --> 00:03:28,447
[Dr. Ada Shannon] In the cache. The usual implementation treats each layer, expert and bit-width as its own object. Expert 17 at 2 bits and expert 17 at 4 bits are unrelated cache entries, so a resident 2-bit copy can't help a later 4-bit request. Storing 1 to 8 bit variants separately would cost 36 bits per weight, nine times one 4-bit copy. And in their simulation, the exact-object hit rate falls from 57.08% for fixed 4-bit to 35.15% for dynamic precision at the same four-bit average. Fewer bytes per request, but far less reuse.

12
00:03:28,447 --> 00:03:32,765
[Hal Turing] So the loss isn't in the bits. It's in what the cache considers the same thing.

13
00:03:32,765 --> 00:04:06,156
[Dr. Ada Shannon] Yes. Their principle is that precision is refinement depth, not a version of the expert. Store one shared state per expert, as ordered refinement units. A precision is a prefix of it. If 2 bits are resident and a request needs 4, you fetch only the missing part. If 6 are resident and a request needs 3, there's no I/O at all. Think progressive JPEG: a base layer plus enhancement layers. The analogy stops at the fact that here the layers are bits of weight precision, which needs custom compute kernels.

14
00:04:06,156 --> 00:04:08,385
[Hal Turing] Is the nested bit representation itself new?

15
00:04:08,385 --> 00:04:42,286
[Dr. Ada Shannon] No, and the framing matters. Matryoshka Quantization, by Pranav Nair and colleagues at Google DeepMind, ICML 2025, trains nested int8, int4 and int2 slices with SGD and distillation. Any-Precision LLM, by Yeonhong Park and colleagues, ICML 2024, is the closest prior work: bitplanes plus nested codebooks, for dense models. So the nested representation is borrowed. What's new is applying it per expert, in an SSD cache, with per-token allocation.

16
00:04:42,286 --> 00:04:43,679
[Hal Turing] And the MoE neighbors?

17
00:04:43,679 --> 00:05:03,881
[Dr. Ada Shannon] SliceMoE, by Choi and colleagues in 2026, does bit-sliced expert caching. HOBBIT and D2MoE also vary expert precision for offloading. HOBBIT keeps precisions as separate cache entries, which is exactly the independent-object design this paper argues against. I'll flag now that I don't see a head-to-head with SliceMoE in the source.

18
00:05:03,881 --> 00:05:10,336
[Hal Turing] So next we should look at how the nesting is built, and how much of that 1.56x is actually reuse.

19
00:05:10,336 --> 00:05:37,782
[Dr. Ada Shannon] Yes. Fewer requested bits is one effect. Sharing cached bits across precisions is another, and only the second is the paper's new idea. Start with the representation, because the tree is what makes sharing possible. For each block of weights, they run K-means to learn 256 values, which is an 8-bit codebook. Then they sort those values and put them at the leaves of a binary tree. Each parent is the frequency-weighted average of its two children.

20
00:05:37,782 --> 00:05:40,243
[Hal Turing] Give me a small one. Say two bits, four values.

21
00:05:40,243 --> 00:06:00,862
[Dr. Ada Shannon] Four sorted codewords get indices 00, 01, 10, 11. The first two share the prefix 0, and their parent is the average of those two, weighted by how many weights landed on each. A weight stored as 01 reads as its own leaf at two bits, and as the parent at one bit. So truncating the index always lands on a valid reconstruction.

22
00:06:00,862 --> 00:06:05,553
[Hal Turing] But the lower levels aren't optimized for anything. They fall out of the 8-bit clustering.

23
00:06:05,553 --> 00:06:19,438
[Dr. Ada Shannon] Right. That's a real limit, and they say so. Their number is about 10% relative weight error at 4 bits, against about 14.8% for GPTQ-INT4. I'd treat that as a practical reference, not a head-to-head.

24
00:06:19,438 --> 00:06:20,553
[Hal Turing] And the bitplanes?

25
00:06:20,553 --> 00:06:38,339
[Dr. Ada Shannon] The stored indices get transposed. Plane one holds the first index bit of every weight in the expert, plane two holds the second, and so on. If you packed each weight's eight bits together, going from three bits to four would mean reading every weight's bits and discarding what you already have. With planes, you read exactly plane four.

26
00:06:38,339 --> 00:06:39,640
[Hal Turing] What does that cost?

27
00:06:39,640 --> 00:06:50,925
[Dr. Ada Shannon] The eight-bit index payload is twice a fixed 4-bit copy. Codebook metadata is under 0.8% of that payload. In exchange, one copy serves anything from one to eight bits.

28
00:06:50,925 --> 00:06:54,918
[Hal Turing] Now the allocator. Why can't it just measure which experts need more bits?

29
00:06:54,918 --> 00:07:19,160
[Dr. Ada Shannon] Because measuring the error means loading the expert, and it has to decide what to load first. So it minimizes a surrogate: each expert's squared routing weight times an offline error profile for that bit-width. It starts every expert at one bit, then repeatedly grants the bit with the biggest gain, until the average-bit budget is spent. With ten experts and seven bit upgrades each, that's only 70 candidates per layer.

30
00:07:19,160 --> 00:07:21,111
[Hal Turing] Why does greedy give valid prefixes?

31
00:07:21,111 --> 00:07:41,823
[Dr. Ada Shannon] Because each extra bit helps less than the one before. 99.45% of profiles already behave that way, and they repair the rest with isotonic regression. Then granting bit four always comes after bit three. One oddity: the profiles come from ten random Gaussian inputs, not task data.

32
00:07:41,823 --> 00:07:43,866
[Hal Turing] Does the surrogate track anything real?

33
00:07:43,866 --> 00:08:03,185
[Dr. Ada Shannon] Against their restricted best-of-candidates estimate, it's close. At a 3-bit budget, output error is about 20% for the allocator and about 30% for uniform. Perplexity at 2.5 bits is 3.240, against 3.304 for the simpler rank-based schedule. It's a modest edge, and it's one metric.

34
00:08:03,185 --> 00:08:04,346
[Hal Turing] Then the runtime.

35
00:08:04,346 --> 00:08:33,278
[Dr. Ada Shannon] A plane is usable only if every earlier plane is resident. When an expert is accessed at b bits, the cache refreshes planes b down to 1, so the deepest planes age out first. If three planes are resident and the request is for five, it reads planes four and five. If five are resident and the request is for three, it reads nothing. Reads for missing planes run while experts that are ready compute. They manage planes themselves rather than using mmap, because the OS page cache has no notion of prefix order.

36
00:08:33,278 --> 00:08:34,532
[Hal Turing] And the compute side?

37
00:08:34,532 --> 00:09:09,966
[Dr. Ada Shannon] An AVX-512 kernel pulls 32-bit slices from the requested planes, assembles 32 indices in a register, looks up the codebook, and feeds BF16 dot products. No BF16 matrix is ever written. Against a materialize-then-multiply path, direct execution is 3.94 to 8.79 times faster. And copying one expert's payload to the GPU takes about 0.114 to 0.121 milliseconds, while the CPU finishes the whole invocation in about 0.095.

38
00:09:09,966 --> 00:09:11,963
[Hal Turing] Okay, the headline. 1.56x.

39
00:09:11,963 --> 00:09:35,833
[Dr. Ada Shannon] It splits into two factors. Uniform 4-bit runs at 5.71 tokens per second. The same dynamic 2.5-bit requests with independent objects reach 6.67. That's 1.17x just from asking for fewer bits. PagedMoE on identical requests reaches 8.89, which is 1.33x over independent. Multiply them and you get roughly 1.56.

40
00:09:35,833 --> 00:09:37,551
[Hal Turing] So the new idea is the 1.33.

41
00:09:37,551 --> 00:10:05,415
[Dr. Ada Shannon] Yes. Read traffic backs it up: SSD reads per token drop from 1.14 to 0.69 GiB, about 40% less for the same requests. GLM5.2 shows the same pattern, 1.34x and 1.25x, combined 1.67x. One caution: Uniform4 here is the paper's own prefix cache reading four planes. It's close to the earlier tuned Fixed4, 5.71 versus 5.65, but not the same system.

42
00:10:05,415 --> 00:10:07,273
[Hal Turing] Does quality survive 2.5 bits?

43
00:10:07,273 --> 00:10:34,579
[Dr. Ada Shannon] On Qwen it mostly does. Grade-school math, GSM8K, scores 96.9 against 97.5 for BF16. The harder GPQA Diamond set has about 198 questions, so one question is half a point, and the 2.5-bit model sits two points below BF16. GLM is worse: on the same math test, about three points below MXFP4. There are no confidence intervals, so I wouldn't read small gaps as ties or wins.

44
00:10:34,579 --> 00:10:36,251
[Hal Turing] And the whole-system comparison?

45
00:10:36,251 --> 00:11:05,508
[Dr. Ada Shannon] They report 3.58x over ik_llama.cpp, 8.89 versus 2.48 tokens per second, at about 29.2 GiB versus 33.27 GiB. But the quantization differs, so that compares whole systems, not the caching idea. The fairer mmap comparison is on the same checkpoint at 32 GiB: 3.61 tokens per second for PagedMoE versus 2.20 for an mmap-based Uniform4, which is 1.64x.

46
00:11:05,508 --> 00:11:13,310
[Hal Turing] So the headline is 1.56x over uniform 4-bit. Before we accept it, what is that uniform 4-bit baseline?

47
00:11:13,310 --> 00:11:45,679
[Dr. Ada Shannon] It's a configuration running on PagedMoE's own prefix cache. It stores eight index bits per weight and reads a four-plane prefix for each request. The optimized fixed-4-bit runtime from the motivation section is a different system. The two land within a hair of each other, at 5.65 and 5.71 tokens per second, but they aren't the same thing. The bigger issue is that most of the bits effect, the 1.17x, would be available to a plain fixed 3-bit or 2.5-bit system with a normal cache. The paper never runs one.

48
00:11:45,679 --> 00:11:50,787
[Hal Turing] But the motivation section did show fixed 3-bit hitting the cache more often than fixed 4-bit.

49
00:11:50,787 --> 00:12:09,363
[Dr. Ada Shannon] It did, at 64.29% versus 57.08%, and that's all we know. There's a hit rate, but no throughput and no quality number for it. So I won't guess its speed. A uniform 3-bit baseline is simply the more honest comparison for the headline, and it's missing.

50
00:12:09,363 --> 00:12:16,004
[Hal Turing] Then the 1.33x. That one is measured against the independent baseline, which reloads the whole prefix on a miss.

51
00:12:16,004 --> 00:12:49,208
[Dr. Ada Shannon] Right, and that baseline is weak by construction. A cheap middle option exists. Let a resident higher-precision copy serve lower-bit requests, and promote it when a request needs more. Section 3.2 describes exactly that, and the experiments never evaluate it. So I read 1.33x as an upper bound on what the nested format buys. The true benefit sits somewhere between 1.0x and 1.33x, and nobody has measured where. That's a hypothesis, not a finding. A replay with downward reuse would settle it.

52
00:12:49,208 --> 00:12:55,478
[Hal Turing] Still, the replay uses identical requests and bit assignments. The reuse effect itself looks real.

53
00:12:55,478 --> 00:13:04,394
[Dr. Ada Shannon] Agreed. The mechanism is cleanly isolated, and the gain shows up on both models. What's unsettled is its size once the obvious cache fix is in place.

54
00:13:04,394 --> 00:13:06,856
[Hal Turing] Where does the novelty sit relative to prior work?

55
00:13:06,856 --> 00:13:37,738
[Dr. Ada Shannon] Not in the representation. Any-Precision LLM already had bitplanes with nested codebooks. The new parts are per-expert paging, the allocator that runs before any I/O, and the SSD cache integration. Matryoshka Quantization takes the other route and trains the nested slices, so every precision is optimized. PagedMoE derives its low levels after the fact from the 8-bit clustering. Whether training-aware nesting would make 1 to 3 bits good enough to go below 2.5 average bits is untested.

56
00:13:37,738 --> 00:13:38,992
[Hal Turing] And the MoE neighbors?

57
00:13:38,992 --> 00:14:00,401
[Dr. Ada Shannon] SliceMoE does bit-sliced expert caching. D2MoE pairs token-adaptive bit-widths with nested weights, and HOBBIT serves low-importance misses at low precision. Those are described but, as far as I can tell, never benchmarked. A head-to-head against SliceMoE-style slicing is the comparison that would back the cross-precision claim.

58
00:14:00,401 --> 00:14:03,791
[Hal Turing] What about quality? You said 2.5 bits holds up on most tasks.

59
00:14:03,791 --> 00:14:36,067
[Dr. Ada Shannon] On most, yes. But it's one run per cell, with no seeds or confidence intervals. GPQA Diamond has about 198 questions, so one question is half a point. Some quantized cells beat the full-precision model, which tells you noise is in play. The GLM math drop of about three points is the one gap too big to wave off. And the quality baselines are GPTQ-INT4 and MXFP4, not a uniform 2.5-bit or 3-bit model, so we can't see what the allocator buys over a plain low-bit alternative.

60
00:14:36,067 --> 00:14:40,246
[Hal Turing] The allocator's sensitivity profiles come from ten random Gaussian inputs, too.

61
00:14:40,246 --> 00:15:00,773
[Dr. Ada Shannon] Yes, and there's no check of those profiles against real task data. The headroom study is also a single layer. The allocator is cache-oblivious as well. It budgets requested bits, not bytes that miss the cache. A cache-aware version could take free bits from resident planes. That's an obvious extension, though untested here.

62
00:15:00,773 --> 00:15:04,256
[Hal Turing] Does any of this touch SSD-based KV cache offloading?

63
00:15:04,256 --> 00:15:16,144
[Dr. Ada Shannon] No. The paper's SSD traffic is expert bitplanes only, and the KV cache stays on the GPU with the dense and attention layers. If both shared one drive, the interaction is unmeasured.

64
00:15:16,144 --> 00:15:17,770
[Hal Turing] So who should care, practically?

65
00:15:17,770 --> 00:15:42,615
[Dr. Ada Shannon] Anyone running a huge MoE on a 64 GB machine at batch size one, where the SSD is the bottleneck. The evidence is one machine, two models and 500 decode tokens from a cold cache. Larger batches are untested, and the 2x index storage is a real cost. Next steps are a tuned fixed-precision baseline, one-directional reuse, training-aware nesting, other SSDs and CPUs, and larger batches.

66
00:15:42,615 --> 00:16:06,392
[Hal Turing] So here's where I land. Cross-precision reuse is a real, measured effect on this setup. It's worth up to 1.33x over a weak independent baseline, on top of about 1.17x from asking for fewer bits. The boundary is that the headline compares against a baseline that isn't the strongest available, and quality at 2.5 bits depends on the model and task.

67
00:16:06,392 --> 00:16:10,804
[Dr. Ada Shannon] That's a fair summary. A solid proof of concept, not yet a settled result.

68
00:16:10,804 --> 00:16:13,312
[Hal Turing] Thanks for listening, everyone. See you next time.