1
00:00:01,000 --> 00:00:43,674
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon. Today we're digging into a paper called STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU. That's Victor J.B. Jung et al., six co-authors total, out of AMD Research and Advanced Development, with ETH Zurich and the University of Bologna also on board. It went up on arXiv on July 10th, 2026. And Ada, when I saw NPU and FlashAttention in the same title I got excited, because this is squarely in the on-device inference space we haven't touched on this show yet.

2
00:00:43,674 --> 00:01:16,674
[Dr. Ada Shannon] Yeah, this one grabbed me for a different reason than the raw speedup numbers, Hal. A lot of hardware papers just say "we ran it and it's fast," and you're left guessing why. This team walks through the dataflow reasoning: why the causal mask creates load imbalance on a fixed pipeline, why that's fundamentally different from a GPU's problem, and how they solve it structurally instead of just throwing more compute at it. That grounded, mechanistic explanation is what convinced me this was worth a full episode instead of a footnote. It's not "look, numbers went up," there's an actual argument for why they went up.

3
00:01:16,674 --> 00:01:29,900
[Hal Turing] So let's set the stage. Why does anyone care about running attention on a laptop chip instead of shipping the whole thing off to a GPU in some data center? Cloud GPUs are stupidly fast.

4
00:01:29,900 --> 00:02:09,600
[Dr. Ada Shannon] Right, but fast isn't the only axis that matters. LLMs are increasingly embedded as agents inside the operating system, reading your files, orchestrating tasks, running continuously in the background. Round-trip every call to a cloud GPU and you've got latency, reliability problems the moment your Wi-Fi hiccups, and a privacy problem, since your file contents are now leaving the device. So chipmakers build NPUs, neural processing units, into laptop SoCs specifically to run inference locally under a tight power and thermal budget. AMD's XDNA is one of these. Keep the model on-device, keep it responsive, don't drain the battery in twenty minutes.

5
00:02:09,600 --> 00:02:20,400
[Hal Turing] Let's define terms, because I don't want to assume everyone remembers the mechanics. What is attention, mechanically, and what problem did FlashAttention solve?

6
00:02:20,400 --> 00:02:59,925
[Dr. Ada Shannon] Attention, from Vaswani et al.'s "Attention Is All You Need" out of Google, 2017, takes three matrices, queries, keys, and values, and computes a weighted sum: multiply queries by keys, scale it, run through softmax to get weights, then multiply those weights by the values. The problem is that intermediate matrix, the scores before softmax, scales with sequence length squared, and naively you'd materialize the whole thing in memory. Tri Dao's FlashAttention, and the FlashAttention-2 follow-up in 2023, fixed that by tiling the computation and using an online softmax trick, building on Milakov and Gimelshein's 2018 work, so you never store the full matrix. Same math, no memory blowup.

7
00:02:59,925 --> 00:03:08,100
[Hal Turing] And STEEL specifically targets the "prefill" stage. What's the difference between that and whatever the other stage is called?

8
00:03:08,100 --> 00:03:29,175
[Dr. Ada Shannon] An LLM runs in two phases. Prefill ingests your entire prompt at once, it's matrix-multiply heavy, and embarrassingly parallel since every token can be processed simultaneously. Decode is the opposite: one token at a time, feeding it back in, over and over, and each step barely does any compute relative to how much memory traffic it needs. More like a trickle than a flood—

9
00:03:29,175 --> 00:03:37,601
[Hal Turing] Wait, sorry, hold on, so prefill is "read the whole book at once," and decode is "write one word, wait, write the next word"?

10
00:03:37,601 --> 00:03:55,701
[Dr. Ada Shannon] Exactly. And STEEL is explicitly only about prefill, reading the whole book. That matters because at long context lengths, prefill can dominate total latency before the model's even said a word back, especially in agentic settings where you're feeding in an entire codebase as context.

11
00:03:55,701 --> 00:04:06,426
[Hal Turing] Now the hardware side. What actually is an NPU, and why is XDNA a "spatial dataflow" architecture? That sounds very different from a GPU.

12
00:04:06,426 --> 00:05:01,426
[Dr. Ada Shannon] It's a different execution model, not just a smaller GPU. A GPU is SIMT: thousands of identical cores run the same instructions on different data, and a cache hierarchy quietly decides what stays close to compute. XDNA is a two-dimensional grid of small VLIW compute tiles, each with its own local scratchpad, wired together by an on-chip network. No cache deciding for you, the programmer or compiler explicitly assigns which tile does which work and schedules every data transfer. That's the "explicit data-movement programming model" the paper talks about. It creates a new problem for attention: because LLMs use a causal mask, each token only sees the tokens before it, so roughly half the attention matrix is zeros. A GPU's scheduler can dynamically shuffle work around that imbalance; on a fixed spatial pipeline, an idle tile just sits idle. That imbalance is a big chunk of what this paper solves.

13
00:05:01,426 --> 00:05:25,651
[Hal Turing] And they solve it well enough to post wild headline numbers: over nine and a half times latency reduction against the prior state of the art, over nine times energy savings against a CPU baseline, and more than twenty-two times speedup against a naive layer-by-layer implementation. We're not unpacking how yet, that's next, but Ada, my curiosity is officially activated.

14
00:05:25,651 --> 00:05:37,551
[Hal Turing] Okay so let's get into the actual machinery, Ada. You said this is a three-stage pipeline. Break that down for me — what physically happens on the chip when a query tile comes in?

15
00:05:37,551 --> 00:06:20,626
[Dr. Ada Shannon] Right, so they take FlashAttention-2's math and split it into three procedures, and critically each procedure lives on its own dedicated AIE core, not just a loop iteration. First stage does the matmul between Q and K to get the raw scores, and it masks and scales them right there. Second stage takes those scores and runs the online softmax update — tracking the running max and the running sum so you never need the full row at once. Third stage takes the softmax output and multiplies it against V to accumulate into the output tile, with a rescale step folded in. Data moves between these three cores through what IRON calls an ObjectFIFO — basically a typed, synchronized queue between tiles. So instead of one core doing everything serially, you've got a literal three-stage assembly line running concurrently.

16
00:06:20,626 --> 00:06:29,901
[Hal Turing] And that's where the causal mask becomes a problem, right? Because if you're autoregressive, half your tiles are just multiplying by zero.

17
00:06:29,901 --> 00:07:19,851
[Dr. Ada Shannon] Exactly, and here's the clever bit. If you assign each pipeline a contiguous chunk of query rows — rows zero through seven to pipeline one, eight through fifteen to pipeline two — the early pipelines get mostly-masked tiles and finish fast, while the late pipelines are doing full unmasked work and become the bottleneck everyone else waits on. Because K and V are broadcast to all pipelines simultaneously, that broadcast can't start until every consumer is ready, so one slow pipeline stalls the whole array. Their fix is to interleave: instead of contiguous chunks, each pipeline gets a scattered set of rows spanning the whole matrix, so every pipeline sees roughly the same mix of masked and unmasked tiles. That single placement change is worth 38 percent latency reduction over the naive layout — no new hardware, just smarter scheduling.

18
00:07:19,851 --> 00:07:29,601
[Hal Turing] Oh wait wait wait — so that 38 percent is purely from rearranging which rows go where? That's basically a free win sitting in the scheduler.

19
00:07:29,601 --> 00:08:08,901
[Dr. Ada Shannon] Pretty much, yeah. But it only works because they'd already solved the harder constraint underneath it, which is Mem-tile port budgeting. Each Mem tile only has six ports, and every STEEL pipeline needs four of them — one for its Q tile, one for output, and two more for swizzling the intermediate P tiles between stage two and three. Q gets distributed, one tile per pipeline, using IRON's distribute primitive; K and V get broadcast to every pipeline since they're shared; and O gets collected back through join. Across ten pipelines that adds up to 42 of the 48 available Mem-tile ports on the chip. That's about as tight as you can pack it — there's genuinely very little headroom left on that resource.

20
00:08:08,901 --> 00:08:17,576
[Hal Turing] So let's get to the numbers, because you teased them last segment but didn't unpack the conditions. What are we actually measuring here?

21
00:08:17,576 --> 00:08:57,202
[Dr. Ada Shannon] Three separate benchmarks, three separate configs, and this matters for interpreting the headline figures. The 22.8x speedup is STEEL versus a layer-by-layer implementation, both on XDNA 2, using BERT's attention shape — 12 heads, head dimension 64. That gain comes from avoiding context-switch overhead between separate GEMM, softmax, and scale kernels, and from never writing the intermediate A and P tensors back to DRAM. They actually quantify that traffic reduction directly: at sequence length 4096, off-chip data movement drops 19.4x, from 9.7 gigabytes down to 0.5 gigabytes, just from fusion avoiding those intermediate round-trips.

22
00:08:57,202 --> 00:09:06,052
[Hal Turing] And the DATO comparison, the 9.6x number — that's a different chip generation and I assume a caveat attached?

23
00:09:06,052 --> 00:09:54,452
[Dr. Ada Shannon] Two caveats, actually. First, that's on XDNA 1, not XDNA 2, and still using the BERT config, not Llama. Second, and this is the one worth flagging clearly: they could only get DATO to compile for sequence lengths up to 4096. Beyond that, compile time apparently blows up exponentially, so the 9.6x average is computed entirely inside that shorter-context regime. Meanwhile the energy numbers — 9.17x over CPU, 1.75x over GPU — use a totally different attention shape, Llama3.1-1B with 32 heads, swept from 2048 all the way to 32768. So when you line up these charts side by side, you're comparing different head counts and different sequence ranges. It's not wrong, but it means you can't just average the headline numbers together as one unified result.

24
00:09:54,452 --> 00:10:03,527
[Hal Turing] One more thing before we move on — this is all prefill, right? What about decode, the token-by-token stage you defined earlier?

25
00:10:03,527 --> 00:11:06,727
[Dr. Ada Shannon] Not tested at all. Every benchmark in this paper — the layer-by-layer comparison, the DATO comparison, the energy sweep — is prefill only. There's no GEMV-style decode kernel here, no decode latency numbers, nothing. That's a real gap given how much of an agentic session's total cost is actually decode after the first turn. Pope and colleagues at Google laid out this exact split back in 2022, in "Efficiently Scaling Transformer Inference" — prefill is compute-bound and matmul-heavy, decode is memory-bound, one query row against a growing KV cache. STEEL's whole pipeline is three big matmul stages feeding each other in a systolic chain. Decode is a GEMV — a single vector times a huge matrix. You can't just point that same three-stage pipeline at a GEMV and expect anywhere near the same utilization. There's a paper, FlashDecoding++, Hong and colleagues, 2024, showing decode needs its own parallelization strategy entirely, even on GPU. Nothing in this paper tells us what a decode kernel costs on an XDNA tile, or whether it's even feasible with this architecture.

26
00:11:06,727 --> 00:11:42,452
[Hal Turing] Right, and that's a real gap given the paper opens by motivating itself with agentic OS workloads specifically. Let's talk about the other big asterisk you flagged earlier — the DATO comparison. That 9.6x number is the headline SotA-beating result. But you said DATO wouldn't compile past sequence length 4096, and the paper's own long-context story runs 2048 to 32768 elsewhere. So is 9.6x actually representative of anything, or is it just the easiest slice of the problem?

27
00:11:42,452 --> 00:12:43,552
[Dr. Ada Shannon] It's the easiest slice. The DATO comparison is entirely confined to the short end of the range the paper cares about elsewhere. STEEL keeps scaling past 4096 in every other experiment — the energy sweep goes to 32768 — but the one comparison against a competing NPU implementation stops exactly where DATO's compiler falls over. That's not necessarily STEEL's fault, DATO's exponential compile blowup is DATO's problem, but it means the 9.6x average is computed over a regime that's structurally favorable and doesn't tell us anything about who wins at 16k or 32k tokens, which is where the paper's own agentic framing lives. And there's a second wrinkle sitting right next to it: the DATO comparison uses BERT, 12 heads, while the energy comparison against CPU and GPU uses Llama3.1-1B, 32 heads. Different configs, different figures — nobody benchmarked the modern-LLM shape against DATO.

28
00:12:43,552 --> 00:12:56,802
[Hal Turing] Hold on, that's actually — so the one config that looks most like a real deployed model, the 32-head Llama shape, never gets compared against the actual competing NPU baseline at all?

29
00:12:56,802 --> 00:13:11,702
[Dr. Ada Shannon] Correct. It only gets compared against CPU and GPU on the same chip, where STEEL obviously wins because it's the native accelerator. The SotA-beating number and the realistic-workload number never touch the same experiment.

30
00:13:11,702 --> 00:13:23,152
[Hal Turing] Okay, so building on that gap — you mentioned GQA earlier as something worth flagging. Walk me through why that matters for STEEL specifically, not just in the abstract.

31
00:13:23,152 --> 00:14:06,127
[Dr. Ada Shannon] Both tested configs, BERT and Llama3.1-1B, use full multi-head attention — equal numbers of query and key/value heads. But production models have moved past that. Ainslie and colleagues at Google, in the GQA paper from 2023, showed you can shrink the KV heads down to a fraction of the query heads with almost no quality loss, and that's now standard in Llama 3 8B and up, Mistral, most serving-grade models. STEEL's whole Mem-tile port budget, that 42-of-48-ports number from Part Two, is built around broadcasting one K and one V tile to every pipeline per head. If you've got eight query heads sharing one KV head instead of a one-to-one mapping, the broadcast pattern and the port math change completely. It might get easier, might get harder, but the paper simply doesn't test the configuration that most new models actually ship with.

32
00:14:06,127 --> 00:14:20,553
[Hal Turing] And that connects to something else that jumped out at me reading the setup — STEEL treats K and V as something you broadcast fresh from DRAM every single call. Real serving stacks don't work that way anymore, do they?

33
00:14:20,553 --> 00:15:00,653
[Dr. Ada Shannon] Not even close. Kwon and colleagues at UC Berkeley, the PagedAttention paper, 2023, is the reason vLLM and basically every serious inference server exists in its current form — you page and reuse the KV cache across turns instead of recomputing or re-fetching it. STEEL benchmarks attention as an isolated kernel with a fresh broadcast every call, and the paper never once discusses how a fused NPU kernel would sit inside, or get bypassed by, a paged KV system in an actual agent runtime. That's not a small omission given the paper's entire motivation is multi-turn agentic sessions on the OS.

34
00:15:00,653 --> 00:15:05,978
[Hal Turing] So stepping back — who should actually care about this paper, practically speaking?

35
00:15:05,978 --> 00:15:47,703
[Dr. Ada Shannon] If you're doing single-shot, first-turn, long-context prefill on a power-constrained laptop — think "summarize this codebase" as a cold start — this is a genuinely strong, honest engineering contribution. The sparsity-aware placement is a real idea, not just a tuning knob, and it's open source on AMD's IRON repo, which lowers the barrier for anyone else doing NPU attention research. Worth noting, Luca Benini's also behind FlatAttention, a similar dataflow-fusion paper on tile-based accelerators, so this is clearly part of a broader research line out of ETH and Bologna, not a one-off. But if your actual workload is a long multi-turn agent loop, this paper answers maybe a third of the question you're asking.

36
00:15:47,703 --> 00:15:50,953
[Hal Turing] What's next, then — what would close the gap?

37
00:15:50,953 --> 00:16:13,478
[Dr. Ada Shannon] A decode-stage NPU kernel that isn't just this pipeline stretched thin, GQA-aware Mem-tile budgeting since that broadcast math needs rework, and some story for integrating with paged KV caching instead of ignoring it. And honestly, rerunning the DATO comparison at 8k or 16k tokens if that compile-time issue ever gets fixed, because right now the SotA claim only holds in the shortest, easiest regime tested.

38
00:16:13,478 --> 00:16:50,828
[Hal Turing] So where that leaves us: STEEL is a real, careful piece of systems engineering — the sparsity-aware placement and the fused pipeline are genuine contributions, and the energy numbers on real hardware are hard to argue with for what they measure. But the paper's framing reaches further than its evaluation does — prefill only, one vendor's chips, MHA only, causal masking only, and a SotA comparison that quietly stops at the point where the baseline still compiles. Know what you're actually getting before you extrapolate to the agentic future it's selling. That's STEEL. Thanks for listening, and we'll catch you next time.