1
00:00:01,000 --> 00:00:09,300
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:09,300 --> 00:00:41,549
[Hal Turing] Today's paper is "98× Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router." The first author is Xunzhuo Liu, with colleagues from the vLLM Semantic Router Project, MBZUAI, McGill, AMD and Red Hat. It's an arXiv paper from March 2026. The claim is that routing an 8K-token prompt drops from 4,918 milliseconds to 50.

3
00:00:41,549 --> 00:00:57,724
[Dr. Ada Shannon] A request that waited almost five seconds now waits a twentieth of a second. That's the headline. What it measures is a pipeline against a pipeline, and the useful work is taking it apart. So we should first say what a semantic router is and why it's squeezed for room.

4
00:00:57,724 --> 00:01:00,999
[Hal Turing] Go ahead. What's the router actually doing?

5
00:01:00,999 --> 00:01:23,399
[Dr. Ada Shannon] It sits inside Envoy, the proxy in front of your LLM servers, as an external processing service called ext_proc. Every request passes through it before any model sees it. The router runs small classifiers on the prompt: is this a jailbreak, does it contain PII, what domain is it. Then it rewrites headers to pick a backend model. All of that happens before the first token, so every millisecond lands directly on time-to-first-token.

6
00:01:23,399 --> 00:01:28,074
[Hal Turing] And the body of the request is the whole prompt, as JSON?

7
00:01:28,074 --> 00:01:49,024
[Dr. Ada Shannon] Yes. Envoy has two relevant modes. In BUFFERED mode it collects the entire body and hands it over as one message, so the router parses the full JSON and serializes it again. In STREAMED mode the body arrives in chunks with an end-of-stream flag. For a few-KB chat message that difference is noise. For a 32K-token prompt it isn't.

8
00:01:49,024 --> 00:01:52,824
[Hal Turing] Why not just give the router its own GPU?

9
00:01:52,824 --> 00:02:13,974
[Dr. Ada Shannon] That's the co-location question. vLLM preallocates nearly all GPU memory for weights and its KV cache, the paged scheme from Kwon and colleagues at Berkeley in 2023. What's left is small. The paper budgets about 718 megabytes per classifier session. The classifier weights are small, so the danger is activations, and attention is where those blow up.

10
00:02:13,974 --> 00:02:17,250
[Hal Turing] Remind me why attention blows up.

11
00:02:17,250 --> 00:02:40,425
[Dr. Ada Shannon] Standard scaled dot-product attention, SDPA, materializes an n-by-n score matrix. In this ONNX export path there's also an n-by-n mask. Per the paper, that mask is about 1.5 gigabytes per session at 8K tokens, so roughly 4.5 gigabytes for three classifiers. It grows fourfold per doubling: 6 gigabytes at 16K, 24 at 32K. Against a 718 megabyte budget, that's fatal.

12
00:02:40,425 --> 00:02:43,500
[Hal Turing] So FlashAttention is the fix.

13
00:02:43,500 --> 00:03:10,849
[Dr. Ada Shannon] Tri Dao and colleagues at Stanford, 2022. It processes the sequence in tiles that fit in on-chip memory and keeps a running softmax, so the full matrix never gets written out. The trick for the running softmax is the online normalizer from Milakov and Gimelshein at NVIDIA in 2018. Rabe and Staats at Google had shown linear-memory attention in 2021. The result is exact, not approximate, and memory drops from quadratic to linear. Compute for full attention is still quadratic, which matters later.

14
00:03:10,849 --> 00:03:14,000
[Hal Turing] If it's four years old, what's new here?

15
00:03:14,000 --> 00:03:37,349
[Dr. Ada Shannon] The plumbing. PyTorch picks a fused kernel for you. ONNX Runtime executes the exported graph node by node, so a fused attention path has to exist as an operator. ONNX Runtime has one for NVIDIA CUDA but not on its ROCm path for AMD, which falls back to plain SDPA. The authors write a custom operator that calls AMD's Composable Kernel library on an MI300X. That's an engineering contribution, not an algorithmic one.

16
00:03:37,349 --> 00:03:43,449
[Hal Turing] Which model are these classifiers? Eight thousand tokens is long for an encoder.

17
00:03:43,449 --> 00:04:11,000
[Dr. Ada Shannon] ModernBERT, from Warner and colleagues at Answer.AI in 2024, and a 270-million-parameter variant called mmBERT-32K stretched to 32K tokens with YaRN. Most layers attend only within a 128-token sliding window, and every third layer attends globally. Sliding windows plus a few global tokens go back to Longformer, Beltagy and colleagues at Allen AI in 2020. Local layers cost roughly linear time. Global layers stay quadratic.

18
00:04:11,000 --> 00:04:15,949
[Hal Turing] So even with the memory fixed, long inputs still get expensive.

19
00:04:15,949 --> 00:04:46,099
[Dr. Ada Shannon] Yes, and that motivates the second stage: shrink the input. Prompt compression is usually done with a neural model. LLMLingua from Microsoft in 2023, LLMLingua-2, Selective Context and RECOMP all fit that pattern. Running another model inside a latency-critical router is awkward. This paper instead scores sentences with classical signals: graph ranking, TF-IDF, position and novelty. It keeps about 512 tokens for the classifiers only. The full prompt still goes to the LLM.

20
00:04:46,099 --> 00:04:48,525
[Hal Turing] Why does position matter?

21
00:04:48,525 --> 00:05:01,600
[Dr. Ada Shannon] Lost in the Middle, Liu and colleagues at Stanford in 2023, found models use the start and end of long contexts better than the middle. The compressor leans on that by favoring both ends.

22
00:05:01,600 --> 00:05:05,800
[Hal Turing] And the third stage is about the proxy, not the model.

23
00:05:05,800 --> 00:05:22,200
[Dr. Ada Shannon] Right. Once classifier cost is roughly constant, JSON handling is a visible slice. So the router decides from the first chunk whether to pass the body through or accumulate it, and it reads fields straight from the bytes instead of unmarshalling the whole thing.

24
00:05:22,200 --> 00:05:32,650
[Hal Turing] So the three stages are FlashAttention to fit the memory, compression to bound the input, and streaming plus zero-copy JSON to trim overhead.

25
00:05:32,650 --> 00:05:44,825
[Dr. Ada Shannon] Yes. Keep the 98× in mind as pipeline-versus-pipeline. How much of it is the attention kernel, and how much is just moving off the CPU, is what we'll try to separate.

26
00:05:44,825 --> 00:05:53,100
[Hal Turing] Let's start with Stage 1. If ONNX Runtime on AMD has no FlashAttention, what did they actually build?

27
00:05:53,100 --> 00:06:29,825
[Dr. Ada Shannon] A graph rewrite plus a custom operator. A Python script scans the exported ONNX graph for the textbook attention pattern: scale the queries, multiply by K transpose, add a mask, softmax, multiply by V. It replaces each match with one node, CKFlashAttention. That node calls AMD's Composable Kernel fused-attention code through a HIP kernel. It's loaded as a 192-kilobyte shared library, compiled for the MI300X's gfx942 architecture, and handles head dimensions 32, 64 and 128.

28
00:06:29,825 --> 00:06:35,675
[Hal Turing] But a fused kernel alone doesn't save memory if the mask is still sitting there.

29
00:06:35,675 --> 00:06:58,650
[Dr. Ada Shannon] Right, and that's the part that matters. The original graph builds a full sequence-by-sequence mask for every batch and head. That mask is the 1.5 gigabytes at 8K from earlier. The rewrite throws it away. It replaces it with a one-dimensional padding bias, one number per token: zero for real tokens, a huge negative for padding. Then it deletes the now-dead subgraph that built the big mask. The kernel computes attention tile by tile, so nothing quadratic is ever stored.

30
00:06:58,650 --> 00:07:02,050
[Hal Turing] And the sliding windows from ModernBERT?

31
00:07:02,050 --> 00:07:22,575
[Dr. Ada Shannon] The script tells local layers from global ones by the names of their mask nodes. Local layers get a window of 63 tokens to the left and 64 to the right, which is the 128-token window. Global layers get minus one on both sides, meaning attend to everything. So the kernel gets a cheap local pattern for two layers out of three and full attention for the third.

32
00:07:22,575 --> 00:07:28,475
[Hal Turing] Fine, so memory is fixed. Stage 2 then attacks the token count itself.

33
00:07:28,475 --> 00:08:01,550
[Dr. Ada Shannon] Yes. Split the prompt into sentences, score each one four ways, keep the best until you hit about 512 tokens, and put them back in original order. The signals are TextRank centrality, a U-shaped position weight where edge sentences count fully and the middle counts half, TF-IDF for rare terms, and a novelty score for sentences unlike the rest of the document. The weights are 0.20, 0.40, 0.35 and 0.05, so position and TF-IDF dominate. The first three and last two sentences are always kept.

34
00:08:01,550 --> 00:08:07,850
[Hal Turing] Novelty gets five percent? I'd expect jailbreak text to be exactly the odd one out.

35
00:08:07,850 --> 00:08:40,525
[Dr. Ada Shannon] That was the idea behind the signal. The authors say heavier novelty weight crowded out sentences that represent the domain, so they turned it down and leaned on the always-keep rule for the edges. We'll come back to what that costs. On speed, the whole thing runs on the CPU with no model call. It takes 2 milliseconds at 2K tokens, 9 at 8K and 19 at 16K. The payoff is that a 16K jailbreak classification drops from 126.6 milliseconds to 10.4.

36
00:08:40,525 --> 00:08:43,950
[Hal Turing] And Stage 3 is about JSON?

37
00:08:43,950 --> 00:09:05,550
[Dr. Ada Shannon] Mostly. A StreamedBodyHandler reads the first chunk and pulls out the model field with gjson, without parsing the whole body. If the client named a model, it passes every chunk through untouched. If the model is "auto", it accumulates chunks, splits sentences as they arrive, and classifies at end of stream. Model rewrites use sjson to edit bytes in place instead of unmarshalling and re-marshalling.

38
00:09:05,550 --> 00:09:08,825
[Hal Turing] Which path did the benchmarks exercise?

39
00:09:08,825 --> 00:09:31,425
[Dr. Ada Shannon] The auto path, since they need classification. The zero-copy passthrough serves explicit-model requests, which the paper doesn't time. Also, the 62 to 50 millisecond gain bundles streaming with the JSON change, and the accumulate path still buffers the whole body. We can't tell which piece earned what. The cleanest evidence is a domain-only setup at 16K tokens, where end-to-end drops from 61 to 31 milliseconds.

40
00:09:31,425 --> 00:09:34,825
[Hal Turing] Now the ladder. Where does the speedup come from?

41
00:09:34,825 --> 00:10:01,775
[Dr. Ada Shannon] The setup first. One MI300X shared with two vLLM instances, ROCm 7.0, ONNX Runtime 1.22.1. Ten sequential curl calls per configuration, two or three runs varying under 3%, on synthetic prompts built so every classifier fires. At 8K tokens, ONNX on CPU takes 4,918 milliseconds. GPU with FlashAttention takes 127. Compression brings it to 62, and streaming to 50.

42
00:10:01,775 --> 00:10:06,850
[Hal Turing] So FlashAttention gives the first 38.7 times.

43
00:10:06,850 --> 00:10:28,050
[Dr. Ada Shannon] The paper's own rows say otherwise. At 512 tokens, plain attention and FlashAttention tie at 19 milliseconds, yet the GPU already beats the CPU by about 36 times end to end. So most of that first step is the GPU move. FlashAttention's job is making 8K fit in the memory budget at all. That split is our inference from their rows, not an ablation they ran.

44
00:10:28,050 --> 00:10:32,175
[Hal Turing] Where does FlashAttention start to pay off by itself?

45
00:10:32,175 --> 00:11:07,575
[Dr. Ada Shannon] Past about 2K tokens. At 4K it's 51 milliseconds against 167 for plain attention, roughly three times faster. At 8K plain attention runs out of memory under the 718 megabyte budget. That's a co-location limit, not a limit of the 192-gigabyte card. Beyond that, FlashAttention's own time grows faster than length: 105 milliseconds at 8K, 259 at 16K, 756 at 32K. Each doubling costs about two and a half to three times more.

46
00:11:07,575 --> 00:11:11,075
[Hal Turing] So memory is linear but time isn't.

47
00:11:11,075 --> 00:11:25,200
[Dr. Ada Shannon] Right. The global layers stay quadratic. The paper never profiles local against global layers, though, so that's the likely cause, not a demonstrated one. It's also why compressing to a fixed 512 tokens matters.

48
00:11:25,200 --> 00:11:28,450
[Hal Turing] And the concurrency and footprint claims?

49
00:11:28,450 --> 00:11:49,650
[Dr. Ada Shannon] Mixed. The 512-to-16K single-request numbers are measured. The 32K entries and every concurrent-load value with compression are derived or estimated. The roughly 60 requests per second at 16K is ten requests divided by 166 milliseconds, arithmetic and not a load test. The footprint claim is under about 800 megabytes.

50
00:11:49,650 --> 00:12:00,450
[Hal Turing] Let me press on the decomposition, because the 38.7 times is the number people will quote. What do we actually know about the split, and what don't we?

51
00:12:00,450 --> 00:12:22,175
[Dr. Ada Shannon] Measured: at about 500 tokens FlashAttention and plain attention tie, yet the GPU is already about 36 times faster. Inferred: that most of the 38.7 is therefore GPU placement. Unknown: plain attention at 8K on an uncontended GPU, because the out-of-memory result comes from the co-location budget, not the MI300X's capacity.

52
00:12:22,175 --> 00:12:30,125
[Hal Turing] And the Candle row at 1,818 milliseconds? Is that a long-context CPU baseline?

53
00:12:30,125 --> 00:12:55,850
[Dr. Ada Shannon] No. It only classifies 512 tokens, and it's slower than the paper's own ONNX CPU at 500 tokens, 803 milliseconds. A CPU row with compression would complete the decomposition, and the paper only estimates that at about 0.8 seconds. Using that estimate, the 98 times is roughly 6 times from compression and CPU-side effects, and 16 times from GPU plus streaming. That's our arithmetic on an unmeasured number.

54
00:12:55,850 --> 00:13:05,000
[Hal Turing] Then compression. Domain accuracy goes from 53.1 to 61.2 percent. Does that show compression helps?

55
00:13:05,000 --> 00:13:28,275
[Dr. Ada Shannon] It shows compression helps a classifier that is struggling on long inputs. The original arm is whole Wikipedia articles from 2K to 16K tokens. A 53.1 percent baseline on eight classes mostly shows the classifier failing at length. The paper doesn't report accuracy on short prompts for either arm. It also has no truncate-to-512 control. So we can't say TextRank, TF-IDF and novelty add anything beyond keeping the first and last sentences.

56
00:13:28,275 --> 00:13:31,125
[Hal Turing] And the other two classifiers?

57
00:13:31,125 --> 00:13:53,450
[Dr. Ada Shannon] Out of every hundred prompts, compression catches about fourteen more PII cases and misses about fourteen more jailbreaks. Those are percentage points, 78.5 to 92.4 and 70.8 to 56.6. The jailbreak drop contradicts the claim of no accuracy loss. Then the paper says that in production, jailbreak and PII run on the full uncompressed prompt, and compression applies to domain routing only.

58
00:13:53,450 --> 00:13:58,000
[Hal Turing] Which would make the 50 milliseconds not the production number.

59
00:13:58,000 --> 00:14:18,275
[Dr. Ada Shannon] Right. The safety path would pay the raw FlashAttention cost, 259 milliseconds per classifier at 16K. The paper doesn't say which configuration the latency tables measured. The PII gain also goes unused in that setup. And even uncompressed, 70.8 percent jailbreak detection is a weak guard.

60
00:14:18,275 --> 00:14:21,400
[Hal Turing] How much weight can that table carry?

61
00:14:21,400 --> 00:14:49,125
[Dr. Ada Shannon] Little. It rests on eight articles. The compression weights were grid-searched on the same 384 cases used for the accuracy report, so there's no held-out split and no confidence intervals. The jailbreak, PII and domain signals are synthetic strings inserted at the start, middle or end. A regex-like Social Security number is easy for a selector to keep. The attacks are also far easier than the ones in JailbreakBench by Patrick Chao and colleagues. I'd want an injection buried mid-prompt in a real RAG context.

62
00:14:49,125 --> 00:14:52,000
[Hal Turing] What about the timing methodology?

63
00:14:52,000 --> 00:15:29,925
[Dr. Ada Shannon] Wall-clock curl, ten sequential requests, averages with no tails. Concurrency above one and everything at 32K are derived or estimated, so the claim of 20 concurrent requests without running out of memory isn't a loaded run. No test shows what the router does to vLLM's own latency. There's also an oddity: the CPU jailbreak row is faster at 8K, 1,519 milliseconds, than at 2K, 1,684. The footprint is odd too. Under 800 megabytes against three 270-million-parameter FP16 models is about 0.54 gigabytes each if weights aren't shared, and the paper doesn't explain it.

64
00:15:29,925 --> 00:15:32,025
[Hal Turing] And the cost table?

65
00:15:32,025 --> 00:15:43,050
[Dr. Ada Shannon] It compares against a dedicated cloud GPU per node, which few operators would buy. A small CPU pool or one shared router GPU isn't costed.

66
00:15:43,050 --> 00:15:45,725
[Hal Turing] Is the attention work new?

67
00:15:45,725 --> 00:16:08,525
[Dr. Ada Shannon] Not as an algorithm, since memory-efficient attention predates FlashAttention. FlashAttention-3, from Jay Shah and colleagues with Tri Dao in 2024, shows how far Hopper kernels go. This is an engineering port that fills an AMD gap in ONNX Runtime, and the paper never compares against peak MI300X attention throughput. Streaming, compression and JSON handling are hardware-agnostic. Stage 1 is AMD-specific.

68
00:16:08,525 --> 00:16:10,575
[Hal Turing] So who should care?

69
00:16:10,575 --> 00:16:29,850
[Dr. Ada Shannon] Operators on AMD GPUs with tight memory who need long-context classifiers. On NVIDIA, take compression and zero-copy JSON, and measure accuracy on your own traffic first. Also check what an explicit model name does to inspection. The paper says that passthrough path skips it.

70
00:16:29,850 --> 00:16:32,750
[Hal Turing] What would settle the open questions?

71
00:16:32,750 --> 00:16:51,150
[Dr. Ada Shannon] A held-out benchmark with short prompts, a truncation baseline and confidence intervals. A real tokenizer for the budget. Streaming ablated from zero-copy JSON. Profiling local against global layers. Tail latency under real traffic, with vLLM running at load.

72
00:16:51,150 --> 00:17:16,875
[Hal Turing] So here's where I land. This is a working open-source way to run long-context classifiers in a small GPU budget, with big latency gains on the authors' pipeline. The 98 times compares whole pipelines, and most of Stage 1 is GPU placement. Compression's accuracy gain is shown only on long synthetic-signal articles, and it costs jailbreak detection. Thanks for listening, everyone.