Raptor: First 3D-DRAM Accelerator for Generative Inference Silicon

Prashant Nair et al. · d-Matrix & University of British Columbia · ISCA 2026 · early silicon
Paper PDF Interactive Visualization
Tags used on every chart: bench silicon simulated illustrative

Decode re-reads the KV cache every token

Llama-70B example: 80 layers, 8 KV heads, head dim ~128, 8-bit. Paper values at 8K / 32K / 128K; slider scales linearly from them.

Three memory options per card

Modeled specs from the paper. SRAM is fast but tiny, HBM is large but starved, 3D-DRAM sits between.

KV read-time floor at the chosen contextIllustrative · derived arithmetic

KV size ÷ bandwidth, KV only (weights not counted). Uses the slider above.

Stack and hierarchy

Logic die (TSMC N4P) bonded face to face under the DRAM die.

Why a 128-byte flit causes overfetch

Step through the bank arithmetic.

Pinless DBI: "stream flipping"

The memory controller inverts; the flag lives as a metadata bit in DRAM, not on a pin.

Toy bus: toggles vs. inversionIllustrative · 256-bit toy, not paper data

Orange cells toggle versus the previous flit. Slide the share of differing bits: random data (50%) barely benefits.

Energy per bit vs. switchingbench silicondashed line & 50% point = episode inference

Measured: 0.455 → 0.376 pJ/bit, read-only, burst 128, 500 MHz, 8 banks. Baseline is the 100%-switching curve.

Bank chainingIllustrative mapping

N=24 regular banks + M=2 spares (purple). Click a physical bank to fail it; muxes rebuild the window.

RowHammer vs. 4 ms refreshanalytical argument

Paper: 200,000 activations × 44 ns row cycle = 8.8 ms > 4 ms. Threshold is an older-node assumption, single aggressor.

Thermalanalytical resistor-network model, not measured

106 W per chiplet. Cooling dominates; the stack adds ~1.5%.

Throughput gain of 3D-DRAMsimulated

Compute fixed at 10 PFLOPS, batch 32 (Poisson-arrival model), minimal-card deployment. No error bars reported.

What is measured, and what is not

What to ask forfrom the episode

References

  1. Raptor: First 3D-DRAM Accelerator for Generative Inference Silicon
  2. Bus-Invert Coding for Low-Power I/O — Stan, Burleson, 1995
  3. Toggle-Aware Compression for GPUs — Pekhimenko et al., HPCA 2016
  4. Fine-Grained DRAM — O'Connor et al., MICRO 2017
  5. Input-Dependent Power Usage in GPUs — Gregersen, Patel, Choukse, 2024
  6. Flipping Bits in Memory Without Accessing Them — Kim et al., 2014
  7. RAIDR: Retention-Aware Intelligent DRAM Refresh — Liu et al., 2012
  8. HMC / HBM3 energy-per-bit disclosures — JEDEC, Micron, 2022-2025
  9. MegaScale-Infer — Zhu et al., 2025
  10. Stratum — Pan et al., 2025
  11. H2-LLM — Li et al., 2025
  12. AI and Memory Wall — Gholami et al., 2024