1
00:00:01,000 --> 00:00:05,690
[Hal Turing] Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:05,690 --> 00:00:35,272
[Hal Turing] Today's paper is "A Full-Stack Performance Evaluation Infrastructure for 3D-DRAM-based LLM Accelerators," from April 2026. It's led by Cong Li at Peking University, with co-authors from Xi'an UniIC Semiconductors, Houmo AI and HKUST. They build a simulator called ATLAS, and they claim it was checked against real silicon. The question is whether a simulator like that can be trusted to steer chip design.

3
00:00:35,272 --> 00:00:46,882
[Dr. Ada Shannon] Almost nobody in the open literature can say they checked against a real 3D-DRAM chip, so that claim matters. We'll need to ask which points were checked. But first the hardware, because the problem is a physical one.

4
00:00:46,882 --> 00:00:51,433
[Hal Turing] Start with why anyone wants this hardware. Why is decoding so hard on a GPU?

5
00:00:51,433 --> 00:01:13,399
[Dr. Ada Shannon] Decoding makes one token per step. To make it, the chip streams essentially all the active weights, plus the growing KV cache, out of memory. That's only a few operations per byte moved, far fewer than a modern GPU needs to stay busy. So the compute sits mostly idle while the memory runs flat out. Prefill is different. It processes the whole prompt in parallel and is compute-heavy.

6
00:01:13,399 --> 00:01:17,904
[Hal Turing] And the KV cache is the per-token keys and values, stored so we don't recompute them.

7
00:01:17,904 --> 00:01:31,418
[Dr. Ada Shannon] Right. Production servers store it in fixed-size blocks that don't have to sit next to each other in memory. That's block-paged KV. It's good for memory management, but it makes DRAM access patterns irregular. That matters later.

8
00:01:31,418 --> 00:01:34,251
[Hal Turing] So HBM is the usual answer to the bandwidth problem.

9
00:01:34,251 --> 00:01:59,282
[Dr. Ada Shannon] HBM stacks DRAM dies with through-silicon vias and sits next to the processor on an interposer. The interface is wide, but the wires are still millimeters long. Edge devices use LPDDR, which has far less bandwidth. The paper says both fall well short of what the compute could use. Their number is a few terabytes per second against hundreds of teraflops.

10
00:01:59,282 --> 00:02:01,000
[Hal Turing] What does hybrid bonding change?

11
00:02:01,000 --> 00:02:40,474
[Dr. Ada Shannon] It bonds DRAM dies to a logic die face to face with copper-to-copper connections, at roughly a three-micron pitch, with no solder bumps. That's far denser than HBM's I/O, and the data path shrinks from millimeters to micrometers. The paper cites access energy of 0.66 to 0.88 picojoules per bit. That's a 77 to 83 percent reduction versus HBM. Multiple DRAM dies stack with mini-TSVs. The logic die is a proper CMOS die, so it can hold matrix engines, vector units, SRAM and a network-on-chip. The stacked DRAM plus that compute is what they call a 3D-Accelerator.

12
00:02:40,474 --> 00:02:42,796
[Hal Turing] How is that different from processing-in-memory?

13
00:02:42,796 --> 00:03:08,152
[Dr. Ada Shannon] Processing-in-memory puts compute inside the DRAM process, which is cramped. Here the compute lives on a normal logic die and talks to DRAM through dense vertical links, so you keep most of an accelerator's programmability. The costs are capacity, which is much smaller than HBM, and heat. DRAM has to stay under 85 degrees Celsius to hold its data, and compute sitting right under it makes that a design constraint.

14
00:03:08,152 --> 00:03:09,453
[Hal Turing] So designs exist already?

15
00:03:09,453 --> 00:03:40,057
[Dr. Ada Shannon] Yes. H2-LLM and a design from Cao and colleagues target the edge. Stratum targets cloud mixture-of-experts. The paper names three gaps. There's no general architecture model, no reusable programming abstraction, and no open evaluation tool. Each design is tuned to its own scenario, and the tools are closed. H2-LLM, for example, models only memory bandwidth and has no vector units, so things like softmax and normalization aren't properly represented.

16
00:03:40,057 --> 00:03:44,097
[Hal Turing] Why not just point Ramulator at it? It's the standard DRAM simulator.

17
00:03:44,097 --> 00:04:05,877
[Dr. Ada Shannon] Ramulator models DRAM standards, not this organization. And a DRAM simulator alone can't tell you what the compute die does, how the network moves data, or how heat limits frequency. LLM accelerator simulators have the opposite problem. They usually treat memory too simply. This paper needs both, plus a thermal model.

18
00:04:05,877 --> 00:04:11,311
[Hal Turing] Before we go further, I want the DRAM vocabulary, because I think this is where the paper gets specific.

19
00:04:11,311 --> 00:04:45,165
[Dr. Ada Shannon] Physical banks form a two-dimensional array in each channel. Together they act as one logical bank per channel. A row of physical banks, concatenated, forms a logical row, and that's the unit you activate and precharge. A bigger logical row means more accesses land on an already-open row. That's row-buffer locality, and it saves the activate and precharge cost. Channel interleaving decides how many consecutive bytes go to one channel before moving to the next. Coarser interleaving helps locality but reduces channel parallelism. Those two pull against each other, and block-paged KV makes the tension sharper.

20
00:04:45,165 --> 00:04:49,438
[Hal Turing] So simulation accuracy depends on getting those details right, not just peak bandwidth.

21
00:04:49,438 --> 00:05:07,642
[Dr. Ada Shannon] Exactly, and that frames my skepticism. A simulator is only as accurate as the points where it was compared with silicon. If the checks cover one chip, conclusions about other channel counts, row sizes or stack depths are extrapolation. Keep that in mind for what follows.

22
00:05:07,642 --> 00:05:11,961
[Hal Turing] Then let's start with the hardware they model. What does the template actually describe?

23
00:05:11,961 --> 00:05:40,197
[Dr. Ada Shannon] A chip made of identical cores. Each core owns a vertical slice of the DRAM stack, with its own memory channels, a matrix engine, a vector engine, an SRAM buffer and a NoC router. The cores sit in a mesh, so reaching another core's data is slower than reaching your own. That's NUMA, and it's why they model communication explicitly. Between chips they use a plain fixed-bandwidth model: link latency plus size divided by bandwidth.

24
00:05:40,197 --> 00:05:42,519
[Hal Turing] And the programming side? You said Triton-like.

25
00:05:42,519 --> 00:06:07,550
[Dr. Ada Shannon] Yes. Inside a core, every core runs the same program on identically shaped shards. That's SPMD, and it keeps load balancing trivial. Between cores it's MPMD, because in something like a ring reduce each core sends to a different neighbor. The primitives are small: declare a tensor in DRAM, allocate a tile in SRAM, copy, gemm, plus split_gemm, split_attention, send and recv.

26
00:06:07,550 --> 00:06:09,082
[Hal Turing] Walk me through one example.

27
00:06:09,082 --> 00:06:41,544
[Dr. Ada Shannon] Their split_gemm example uses eight cores as a two-by-four grid, multiplying a 16-by-1024 matrix by a 1024-by-1024 one. The inner dimension is split across the two rows, and the output columns across the four columns. So each core does a 16-by-512 times 512-by-256 piece, using the ordinary tiled matmul loop: copy tiles in, gemm, accumulate, write out. The cores end up with partial sums, which a communication kernel then reduces.

28
00:06:41,544 --> 00:06:44,795
[Hal Turing] So the simulator reads that program. What happens next?

29
00:06:44,795 --> 00:07:18,649
[Dr. Ada Shannon] Four stages. A thermal analyzer, built on HotSpot 7.0, checks the temperature and lowers the clock until DRAM is under its 85 degree retention limit. An operator parser and tiling explorer choose tile sizes, with optional autotuning. Then a cycle-level simulator runs the result. The DRAM side extends Ramulator2 with the industry partners' timing parameters and a tile-level command scheduler. The NoC is BookSim2, and compute is a model of FLOPs and SRAM traffic. The whole thing is about 5.4 thousand lines of C++.

30
00:07:18,649 --> 00:07:22,597
[Hal Turing] Now the part you were skeptical about. What did they check against silicon?

31
00:07:22,597 --> 00:08:00,770
[Dr. Ada Shannon] One test chip: four DRAM dies, sixteen cores, sixteen channels per core, half a gigabit per pin, 7 nanometer logic. They collected distinct memory traces and operator shapes from production LLM workloads. That gives 2,304 DRAM cases and 712 compute and communication cases. Typical error is a few percent: about 4% on DRAM latency and bandwidth, about 2% on compute, under 3% on core-to-core communication. The worst single case is 8.57%, and end-to-end error is within 6.37%.

32
00:08:00,770 --> 00:08:03,742
[Hal Turing] That's a real number against real silicon. What's missing?

33
00:08:03,742 --> 00:08:29,238
[Dr. Ada Shannon] The paper doesn't say whether 6.37% comes from a measured full-model run or from composing the per-operator errors. It reports no error bars and no held-out workloads. There's no second chip. For thermal, only the material parameters were measured, using electron microscopy and thermoreflectance. The paper doesn't say the temperature predictions were compared with the chip. I'll return to that. First, what they did with the simulator.

34
00:08:29,238 --> 00:08:31,003
[Hal Turing] Start with the memory organization.

35
00:08:31,003 --> 00:09:08,062
[Dr. Ada Shannon] Interleaving first. Bigger chunks per channel give more row hits but spread the traffic over fewer channels. Utilization rises up to a moderate granularity, then GEMM flattens and attention gets worse. So the middle wins. Channel count follows a similar pattern. Going from two to sixteen channels helps GEMM a lot, by up to 47 percentage points. Beyond sixteen, moderate-sized GEMM tiles can't keep all the channels busy, and utilization drops by as much as 39 points. Logical row size is the easy one. Bigger rows keep helping GEMM, by up to about 61 points, so they land on 64KB.

36
00:09:08,062 --> 00:09:12,520
[Hal Turing] So more channels eventually hurt even before heat enters. Does heat make it worse?

37
00:09:12,520 --> 00:09:49,300
[Dr. Ada Shannon] Yes. Heat is the real constraint on the accelerator side. They fix the core count and a 1,024-pin budget, then vary channels per core. Cutting from sixteen channels to four removes 75% of the bandwidth and buys back only 7.3% more compute. Going the other way costs far more. At 32 channels, compute drops 30% just to scrape under the limit, at 84.9 degrees. At 64 channels, even the lowest clock and smallest compute reach 110 degrees. That assumes a liquid cooling plate, which matters.

38
00:09:49,300 --> 00:09:51,204
[Hal Turing] What did the rest of the sweep settle on?

39
00:09:51,204 --> 00:10:21,855
[Dr. Ada Shannon] Four megabytes of SRAM, because shrinking to one megabyte saves only about 11% area. A matrix-to-vector ratio of 32 to 1, and 128-byte NoC links. The workloads were OPT-66B, LLaMA3-70B, Mixtral-8x22B and Qwen3-235B-A22B at batch 16 and 64, decode only. The order matters. They fixed bandwidth first, then each later dimension given the earlier choices. It's a greedy sequence, not a joint search.

40
00:10:21,855 --> 00:10:27,613
[Hal Turing] So that 64-channel result, and the 64KB rows, sit well outside the one chip they checked.

41
00:10:27,613 --> 00:11:04,347
[Dr. Ada Shannon] Right, and that's the central boundary. The accuracy is real, but it was measured at one chip. The conclusions that matter most, 16 channels, 64KB rows, the thermal wall at 64 channels, sit away from that point. Nobody has shown the timing and controller models hold there. Two things would settle it. One is a second silicon configuration with a different channel count or row size. The other is a ranking check: measure two designs on the chip, and see whether the simulator orders them correctly. Design-space exploration needs the ordering to be right, and the paper reports per-case error instead.

42
00:11:04,347 --> 00:11:08,713
[Hal Turing] Okay, now the headline comparison. What did the derived design actually win?

43
00:11:08,713 --> 00:11:28,728
[Dr. Ada Shannon] About two and a half times the speed of an H200 on average, and about 6.7 times the energy efficiency. The abstract says up to 3.64 times, but that's the best case. Against Stratum, the cloud design from earlier, the average gain is only 8%. And on small-batch mixture-of-experts, Stratum wins by up to 39%.

44
00:11:28,728 --> 00:11:31,468
[Hal Turing] How did they get Stratum numbers? Did they run it?

45
00:11:31,468 --> 00:11:55,385
[Dr. Ada Shannon] No. They emulate it inside ATLAS from its published parameters. So the gap depends on how faithfully the authors reproduced a competitor, using their own simulator. They also reserved extra network bandwidth and SRAM per core for it, so I can't say how much of the dense-model win comes from architecture versus extra resources. And the paper reports no uncertainty on an 8% margin, so I wouldn't call it a clear win.

46
00:11:55,385 --> 00:11:56,778
[Hal Turing] And the GPU side?

47
00:11:56,778 --> 00:12:15,586
[Dr. Ada Shannon] The H200 comparison is at equal area with assumed specs, not a measured GPU run in the source. So the 2.5x is a simulated comparison against a modeled GPU. The paper also doesn't account for cooling cost, which matters given how much of the design is shaped by heat.

48
00:12:15,586 --> 00:12:21,762
[Hal Turing] What about the edge results? They extend H2-LLM, which shares authors with this paper.

49
00:12:21,762 --> 00:12:47,212
[Dr. Ada Shannon] Yes, so the baseline comes from the same group. The extension adds vector units and an 8-to-1 matrix-to-vector ratio, and gets about 1.08 times the speed and 1.09 times the energy efficiency. They also find fine-grained interleaving suits small edge workloads, which is sensible. But it's a small gain, also from simulation, so it doesn't carry the claim that ATLAS guides design across scenarios.

50
00:12:47,212 --> 00:12:50,230
[Hal Turing] Other gaps I noticed: thermal, and what's actually open.

51
00:12:50,230 --> 00:13:22,552
[Dr. Ada Shannon] The thermal model uses measured material properties, but the paper shows no predicted-versus-measured chip temperature. The conclusions also assume liquid cooling at an optimistic heat-transfer coefficient. Under air or ordinary datacenter cooling, the channel and compute split could change. The sweep is greedy, one parameter at a time, so the order could matter. It's decode only, on a fixed 4-by-4 mesh. The code is promised on publication, not released. And the DRAM timing parameters come from industry partners, so outsiders can't inspect them.

52
00:13:22,552 --> 00:13:27,986
[Hal Turing] What about circularity? The validation traces and the exploration workloads come from the same place.

53
00:13:27,986 --> 00:13:50,416
[Dr. Ada Shannon] That's my concern. Validation covers tiled matrix multiplies and block-paged KV. No held-out check is reported for irregular expert gathers, other KV block sizes, or odd shapes. And the paper doesn't say whether the end-to-end error came from a measured full-model run or was composed from per-operator errors. So it's one chip, and we can't tell how well the checks generalize.

54
00:13:50,416 --> 00:13:53,110
[Hal Turing] Fair enough, though one chip is still more than most get.

55
00:13:53,110 --> 00:14:06,763
[Dr. Ada Shannon] Agreed. Most 3D-DRAM work shows no silicon at all. It's a credible template, and the direction of the takeaways holds up. Bandwidth is bounded by area and heat. Prefill is also unaddressed, since they study decode only.

56
00:14:06,763 --> 00:14:08,017
[Hal Turing] So who should care?

57
00:14:08,017 --> 00:14:23,156
[Dr. Ada Shannon] People designing or evaluating 3D-DRAM accelerators. Use it for the shape of the tradeoffs and for sanity-checking a design near the validated chip. Treat the absolute numbers far from it as hypotheses until there's a second silicon point.

58
00:14:23,156 --> 00:14:42,986
[Hal Turing] Then here's where I land. ATLAS gives the field a usable, silicon-checked way to model these chips, and it shows that more bandwidth runs into area and heat. The boundary is that accuracy was shown at one chip. The trends beyond it, and the margins over Stratum, remain unverified. Thanks for listening, and we'll see you next time.