1
00:00:01,000 --> 00:00:07,408
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,408 --> 00:00:41,124
[Hal Turing] Today's paper is "Early Silicon of Raptor: The First 3D-DRAM Accelerator for Generative Inference," from Prashant Nair and colleagues at d-Matrix and the University of British Columbia. It's an ISCA 2026 paper. The question is whether you can build a logic-on-3D-DRAM inference card with about 100 terabytes per second of memory bandwidth as a real product. That means redesigning channel mapping, bus power, yield and thermals around KV-cache traffic. And what does the early silicon actually show?

3
00:00:41,124 --> 00:00:52,641
[Dr. Ada Shannon] The answer is mixed, and the mix matters. Some numbers come from chips on a bench, and some come from simulation. We'll keep those apart as we go. First, why anyone wants this memory at all.

4
00:00:52,641 --> 00:00:56,774
[Hal Turing] Right, so start with decode. Why is generating text memory-bound?

5
00:00:56,774 --> 00:01:26,077
[Dr. Ada Shannon] Prefill processes your whole prompt at once, so the weights get reused across thousands of tokens. That phase is compute-bound. Decode produces one token at a time, and each token depends on the previous one, so you can't batch across time. Every step re-reads the model weights and the entire KV cache, the stored keys and values for all earlier tokens. Little gets reused, so the speed limit is how fast bytes arrive, not how fast you multiply.

6
00:01:26,077 --> 00:01:27,424
[Hal Turing] And the KV cache gets big.

7
00:01:27,424 --> 00:01:51,109
[Dr. Ada Shannon] The paper's Llama-70B example: 80 layers, 8 KV heads, head dimension around 128, stored in 8 bits. That's about 0.16 megabytes per token, per user. At 8K context, roughly 1.25 gigabytes. At 32K, about 5. At 128K, about 20 gigabytes, for one user. And all of it gets read again for every new token.

8
00:01:51,109 --> 00:01:54,266
[Hal Turing] So that's a bandwidth floor before you've even touched the weights.

9
00:01:54,266 --> 00:02:16,233
[Dr. Ada Shannon] Yes. Their Fig. 2 plots that lower bound. Two metrics to hold onto: TPOT is time per output token, lower is better. Interactivity is its inverse, one over TPOT. Tokens per second per card is steady-state throughput for one accelerator. The paper plots one against the other.

10
00:02:16,233 --> 00:02:17,579
[Hal Turing] Then the memory options.

11
00:02:17,579 --> 00:02:48,090
[Dr. Ada Shannon] The paper models three. SRAM: about 150 terabytes per second, but only 4 gigabytes per card. HBM, at an HBM4-class 18 terabytes per second and 192 gigabytes. Raptor's 3D-DRAM: 100 terabytes per second and 32 gigabytes. SRAM is fast but tiny, so big models spread across many cards and pay for communication. HBM holds the model but starves it. 3D-DRAM is the bet on a middle point.

12
00:02:48,090 --> 00:02:49,994
[Hal Turing] What makes it a middle point physically?

13
00:02:49,994 --> 00:03:11,728
[Dr. Ada Shannon] The DRAM die sits directly on the logic die. They bond face to face, through a 36-micron-pitch microbump array. Dense, short vertical links replace HBM's long, fast, power-hungry interface. Each bump carries one bit per cycle, with no multi-cycle bursts. That detail causes a problem.

14
00:03:11,728 --> 00:03:12,518
[Hal Turing] Which is?

15
00:03:12,518 --> 00:03:41,403
[Dr. Ada Shannon] Data bus inversion. Stan and Burleson's bus-invert coding, from 1995, says that if more than half the wires would toggle compared with the previous bus state, send the inverted word plus one flag bit. That caps toggles at N over 2. The catch is that on random wide words, the average saving is small. Commodity DRAM applies it per beat of a burst, with a dedicated flag pin. A single-cycle interface has no beats, and a flag bump per byte costs scarce area.

16
00:03:41,403 --> 00:03:43,586
[Hal Turing] So the headline here is pinless DBI.

17
00:03:43,586 --> 00:03:53,199
[Dr. Ada Shannon] Right. The paper's term is stream flipping. The memory controller does the inversion, and the flag lives as a metadata bit in the DRAM instead of on a pin. We'll get to the mechanism.

18
00:03:53,199 --> 00:03:56,125
[Hal Turing] Why would toggles matter so much if the wires are short?

19
00:03:56,125 --> 00:04:13,308
[Dr. Ada Shannon] Pekhimenko and colleagues' 2015 work on toggle-aware compression for GPUs is the useful precedent. Compression cuts bytes moved but can raise bit toggles, because compressed data looks random. Link energy depends on bit patterns, not just byte counts. Keep that in your pocket.

20
00:04:13,308 --> 00:04:15,258
[Hal Turing] The abstract lists four challenges.

21
00:04:15,258 --> 00:04:52,642
[Dr. Ada Shannon] Mapping the workload onto channels. Power without burst DBI. Resilience, since high bank counts mean some banks will be bad. And heat. For the middle two, the paper uses a few terms. A flit is 128 bytes, delivered per channel to one tensor engine. Bank chaining puts spare banks among regular ones, and muxes pick a contiguous working window so channel width never changes. And RowHammer, from Kim and colleagues at ISCA 2014: repeatedly activating one DRAM row flips bits in neighboring rows. Refreshing fast enough can outrun it, which is where the thermal-aware refresh argument comes in.

22
00:04:52,642 --> 00:04:54,964
[Hal Turing] Before we go further, what should listeners watch?

23
00:04:54,964 --> 00:05:11,172
[Dr. Ada Shannon] One question. The headline energy saving from pinless DBI is about 18 percent. Measured against what baseline? Hold that thought. And the split: throughput comparisons are simulation. Power, bandwidth and refresh come from silicon.

24
00:05:11,172 --> 00:05:17,302
[Hal Turing] Okay, let's open the box. How is the card organized, and why does a 128-byte flit cause trouble?

25
00:05:17,302 --> 00:05:40,615
[Dr. Ada Shannon] Hierarchy first, briefly. A chiplet holds gangs, gangs hold slices, and each slice has 16 channels, so 256 channels per chiplet. The logic die is TSMC N4P, bonded face to face onto the DRAM. A multi-chip module is rated at 422 watts, with a 105 degree junction limit. The trouble is the arithmetic.

26
00:05:40,615 --> 00:05:41,172
[Hal Turing] Go on.

27
00:05:41,172 --> 00:06:03,556
[Dr. Ada Shannon] A flit is 1024 bits, and each bank delivers 256 bits. So a channel wants four banks. But the die only has 840 banks, and 72 are held back as spares. That leaves 768, which is three banks per channel. Three banks give 768 bits, so one 128-byte flit takes two accesses.

28
00:06:03,556 --> 00:06:06,389
[Hal Turing] And the second access drags in data you didn't ask for.

29
00:06:06,389 --> 00:06:41,404
[Dr. Ada Shannon] Right, overfetch. The naive fix is staggering columns across banks, but that needs a 192-byte shifting buffer. Stream blocking avoids it. Each flit is split into a fixed 96-byte aligned part and a 32-byte partial region. The tile layout makes that work. In the Llama-70B example, a tile is 16 kilobytes, which is 128 flits, and 124 columns per row. That matches paged-attention page sizes. The payoff is 2.5 nanoseconds per flit, and about 105 terabytes per second per card at 700 megahertz.

30
00:06:41,404 --> 00:06:45,120
[Hal Turing] So the KV layout was designed around the memory, not the other way round.

31
00:06:45,120 --> 00:07:04,810
[Dr. Ada Shannon] Yes. Now the power problem. I/O costs about 0.45 picojoules per bit. At 100 terabytes per second, that's roughly 360 watts per card, just moving bits. So you need something like DBI. But the single-cycle interface has no burst to hide a flag pin in. Stream flipping moves the job into the memory controller.

32
00:07:04,810 --> 00:07:06,064
[Hal Turing] Mechanism first, please.

33
00:07:06,064 --> 00:07:26,358
[Dr. Ada Shannon] On a write, the controller compares the new flit with the previous flit written on that channel. If inverting it would cut transitions, it stores the inverted data. One metadata bit per flit records that choice, kept in DRAM next to the ECC. On a read, the controller fetches the bit and undoes the inversion. No extra pin and no PHY change.

34
00:07:26,358 --> 00:07:27,705
[Hal Turing] And the measured saving?

35
00:07:27,705 --> 00:07:51,389
[Dr. Ada Shannon] The silicon shows effective switching dropping to 40 to 48 percent. Energy falls from 0.455 to 0.376 picojoules per bit, about 18 percent. That's read-only, burst length 128, 500 megahertz, with eight banks active. Notice the shape. Over half the switching disappears, but the energy falls by less than a fifth.

36
00:07:51,389 --> 00:07:53,711
[Hal Turing] So most of the per-bit energy isn't toggling.

37
00:07:53,711 --> 00:08:09,733
[Dr. Ada Shannon] If energy is roughly linear in switching, about two thirds of it is independent of toggles. That's my inference, not theirs. An 18 percent cut is about 65 watts, and only if the card really streams at full rate. I'd like to know what that's measured against, but that's for later.

38
00:08:09,733 --> 00:08:14,516
[Hal Turing] Fine, parking it. Next challenge: yield. With this many banks, some will be dead.

39
00:08:14,516 --> 00:08:41,126
[Dr. Ada Shannon] Bank chaining. Spare banks are interleaved with regular ones, and muxes on the logic die pick a contiguous window of working banks. With N regular banks and M spares, say N of 24 and M of 2, a channel still sees the same width even when a bank fails. Fig. 11 shows yield for this, but without absolute numbers, so you see the trend, not the yield.

40
00:08:41,126 --> 00:08:45,538
[Hal Turing] What about heat? Stacked DRAM above 85 degrees needs faster refresh.

41
00:08:45,538 --> 00:09:04,439
[Dr. Ada Shannon] Here the deep banking helps. Banks have 1364 rows, 16 to 32 times fewer than commodity DRAM. Refreshing every 4 milliseconds costs only about 1.4 percent of bandwidth. The paper pairs that with interleaved Reed-Solomon [144,140] ECC, co-located with the DBI bit.

42
00:09:04,439 --> 00:09:05,322
[Hal Turing] And RowHammer?

43
00:09:05,322 --> 00:09:26,545
[Dr. Ada Shannon] Their argument: hammering a neighbor 200,000 times at a 44 nanosecond row cycle takes 8.8 milliseconds. Every row is refreshed in 4. Two caveats. The 200K threshold is an assumption for an older node, and it assumes one aggressor row. Double-sided patterns would need fewer activations per row. The paper doesn't discuss that.

44
00:09:26,545 --> 00:09:28,217
[Hal Turing] Thermal results next. Measured?

45
00:09:28,217 --> 00:09:46,049
[Dr. Ada Shannon] No, an analytical resistor-network model. Cooling is about 80 percent of the thermal resistance. The stack adds about 1.5 percent. Hotspots reach about 92 degrees at 106 watts per chiplet, and the DRAM runs about 3.5 degrees cooler than the logic.

46
00:09:46,049 --> 00:09:48,325
[Hal Turing] Now the system results. Silicon or simulation?

47
00:09:48,325 --> 00:10:05,461
[Dr. Ada Shannon] Simulation. They fix the compute at 10 petaflops and swap the memory between SRAM, HBM and 3D-DRAM with the specs we just heard. Batch size 32 comes from a Poisson-arrival model. In the minimal-card deployment, each substrate uses as few cards as the model fits on.

48
00:10:05,461 --> 00:10:06,483
[Hal Turing] And the result?

49
00:10:06,483 --> 00:10:20,833
[Dr. Ada Shannon] At 4K context, 3D-DRAM produces about four and a third tokens for every one from HBM, and about three for every one from SRAM. The abstract's 4.71 and 2.44 are averages over broader settings.

50
00:10:20,833 --> 00:10:22,876
[Hal Turing] Why does SRAM lose on big models?

51
00:10:22,876 --> 00:10:49,672
[Dr. Ada Shannon] It needs many cards just to hold the weights, and that inflates communication. HBM holds the model but can't feed it fast enough. Sensitivity to network latency was low. Two exceptions matter, though. On dense models, SRAM and baseline 3D-DRAM curves are similar. On the speech models, Whisper and Canary, with 448-token contexts on one card, SRAM wins. Extra capacity alone doesn't help there.

52
00:10:49,672 --> 00:10:50,926
[Hal Turing] Any variance reported?

53
00:10:50,926 --> 00:11:03,093
[Dr. Ada Shannon] No error bars. And Table II lists Llama-70B as one 32-gigabyte card, while the text says tensor parallelism of 4. I can't resolve that from the paper, so I won't build on it.

54
00:11:03,093 --> 00:11:05,090
[Hal Turing] Is there related work on the same idea?

55
00:11:05,090 --> 00:11:25,477
[Dr. Ada Shannon] The paper compares against H2-LLM and Stratum, both simulation-based designs. That's the useful context: those are projections, while Raptor has silicon behind its bandwidth, power and refresh numbers. Its throughput gains, though, are still simulated, and bandwidth ratios across different simulators and target workloads aren't apples to apples.

56
00:11:25,477 --> 00:11:30,261
[Hal Turing] Earlier I asked what baseline the 18 percent is measured against. Ada, what did you find?

57
00:11:30,261 --> 00:11:46,515
[Dr. Ada Shannon] The worst case. The curve at one hundred percent switching, where every bit toggles on every cycle. Stream flipping brings traffic down to 40 to 48 percent switching, and the paper compares against that all-toggle ceiling. It shows no unencoded run on actual tensors.

58
00:11:46,515 --> 00:11:49,162
[Hal Turing] But nobody's data toggles every bit every cycle.

59
00:11:49,162 --> 00:12:14,797
[Dr. Ada Shannon] Random data toggles about half its bits. And bus-invert coding caps toggles at half the word. On a 1024-bit flit, random words rarely exceed 512 flips, so the coding barely triggers. Stan and Burleson's scheme saves under three percent there. So landing at 40 to 48 percent means the data is structured. That's interesting, but the paper never says which tensors, which datatypes, or which layers.

60
00:12:14,797 --> 00:12:16,979
[Hal Turing] Can we estimate what the honest number would be?

61
00:12:16,979 --> 00:12:39,549
[Dr. Ada Shannon] This is my inference, not the paper's. Assume energy is linear in switching. Then an unencoded stream at fifty percent sits near 0.385 picojoules per bit, against the measured 0.376. That's a saving of about two percent. Call it a few percent, with wide uncertainty. Against roughly 360 watts of I/O, that's single-digit watts, not 65.

62
00:12:39,549 --> 00:12:40,803
[Hal Turing] And at the card level?

63
00:12:40,803 --> 00:13:05,416
[Dr. Ada Shannon] Murkier. The paper says the 1.1 volt array rail is 87 percent of active power, which sits awkwardly beside a 360-watt I/O estimate on a 422-watt module. The abstract says memory-subsystem energy, while Fig. 10 shows I/O energy at eight banks. No tokens per joule is reported, though that's the motivating metric.

64
00:13:05,416 --> 00:13:07,552
[Hal Turing] Then there's the read-versus-write problem.

65
00:13:07,552 --> 00:13:41,871
[Dr. Ada Shannon] Right. The encoder decides at write time, against the previous flit written on that channel. The toggles that cost energy happen on reads, between flits that are consecutive on the bus. Fig. 10 is read-only, so it can't tell us how often those orders match. Sequential weight tiles probably preserve it. MoE expert gather breaks the chain at every tile boundary. Paged-attention eviction rewrites flits whose neighbors were encoded against old data. Tail appends write one flit at a time. And the partial region is read before the aligned part, so bus order isn't write order.

66
00:13:41,871 --> 00:13:43,265
[Hal Turing] Does that mean it fails?

67
00:13:43,265 --> 00:13:47,676
[Dr. Ada Shannon] No. It means unknown. The paper reports nothing there, so I won't claim it breaks.

68
00:13:47,676 --> 00:13:51,717
[Hal Turing] Datatypes seem like the bigger swing. Quantized weights are denser.

69
00:13:51,717 --> 00:14:14,890
[Dr. Ada Shannon] That's the Pekhimenko result again. INT4 or FP8 values look closer to random, which pushes switching toward fifty percent and shrinks any gain. BF16 has structured sign and exponent bytes. The paper doesn't say which format produced Fig. 10. And power density then depends on the data, while the thermal model assumes a fixed 106 watts per chiplet.

70
00:14:14,890 --> 00:14:19,441
[Hal Turing] There's also the metadata bit itself. One flipped bit inverts a whole flit.

71
00:14:19,441 --> 00:14:41,779
[Dr. Ada Shannon] That worries me more than the energy. I can't tell from the text whether Reed-Solomon covers the stored bits or the logical data, or whether the flag is protected at all. Also, one bit per 1024 is about 0.1 percent capacity, but Fig. 7 says 0.8 percent. The text doesn't reconcile that, and it doesn't say what fetching the flag ahead of the data costs the 100 terabytes per second.

72
00:14:41,779 --> 00:14:43,172
[Hal Turing] What about the rest of the evidence?

73
00:14:43,172 --> 00:15:12,801
[Dr. Ada Shannon] The throughput numbers are simulation, with no error bars. The HBM baseline is a modeled 18 terabyte-per-second part. Minimal-card deployment gives each memory a different card count, and tokens per card isn't tokens per watt or per dollar. The RowHammer argument assumes one aggressor at a 200K threshold from an older node. Yield is shown without absolute numbers, so 72 spare banks out of 840 can't be weighed. And this is early silicon from the vendor selling it.

74
00:15:12,801 --> 00:15:16,051
[Hal Turing] If someone's evaluating this, what should they ask for?

75
00:15:16,051 --> 00:15:37,785
[Dr. Ada Shannon] Encoded versus unencoded energy on real weights and real KV traces, per datatype. Read-order traces from paged attention and MoE gather. Tokens per joule at the card. Beyond that, the authors point to heterogeneous setups, like attention-FFN disaggregation or speculative drafts on Raptor, and to two-high and four-high stacks.

76
00:15:37,785 --> 00:15:59,101
[Hal Turing] So, the takeaway. This establishes a silicon-validated 3D-DRAM interface near 100 terabytes per second, cheap 4 millisecond refresh, and a plausible resilience scheme, plus modeled throughput gains. The boundary is that the pinless-DBI saving is shown only against a worst-case curve, so the real-traffic saving is still unmeasured.

77
00:15:59,101 --> 00:16:00,448
[Dr. Ada Shannon] That's it. Thanks for listening.

78
00:16:00,448 --> 00:16:02,445
[Hal Turing] Thanks, everyone. See you next time.