1
00:00:01,000 --> 00:00:09,034
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:09,034 --> 00:00:50,086
[Hal Turing] Today's paper is "Cascade: Exploiting SLO-Aware Latency Budget for Fair and High Goodput LLM Inference Serving," by Muhammad Adnan and colleagues. Adnan did the work as an intern at Microsoft Azure Research, and the affiliations are the University of British Columbia, Microsoft Azure Research and NVIDIA. It appeared on arXiv in August 2026. The question: if you subtract a request's predicted service time from its latency target, can that one leftover budget drive both the order of the queue and the decisions about restoring KV cache, without starving long prompts?

3
00:00:50,086 --> 00:01:11,727
[Dr. Ada Shannon] Same latency target, very different urgency. That's the problem. One cluster now serves chat, tool and agent calls, coding and reasoning. Inside a single class, prompts and outputs can differ by orders of magnitude, and so can how much cache is already warm. A scheduler that treats them as equals wastes capacity.

4
00:01:11,727 --> 00:01:15,303
[Hal Turing] Let's set the vocabulary. Start with the two phases of inference.

5
00:01:15,303 --> 00:01:38,477
[Dr. Ada Shannon] Prefill reads the whole prompt in one compute-heavy pass and builds the KV cache, the per-token attention state. Decode then emits tokens one at a time and is limited by memory bandwidth. Each phase gets its own latency target. Time to first token, TTFT, covers prefill. Time per output token, TPOT, covers decode. Lower is better for both.

6
00:01:38,477 --> 00:01:39,406
[Hal Turing] And goodput?

7
00:01:39,406 --> 00:02:01,093
[Dr. Ada Shannon] Throughput counts every token served. Goodput counts output tokens per second only from requests that meet both the TTFT and TPOT targets. A server can look busy and still deliver little goodput if most answers arrive late. DistServe, from Zhong and colleagues in 2024, popularized that framing for LLM serving.

8
00:02:01,093 --> 00:02:06,201
[Hal Turing] Now the scheduling failures. Take one 20,000-token prompt sitting ahead of many short chats.

9
00:02:06,201 --> 00:02:38,756
[Dr. Ada Shannon] First-come-first-served, FCFS, is the vLLM default. It never starves anyone, but that big prefill blocks everything behind it. That's head-of-line blocking. Shortest-job-first fixes it by serving short work first, which helps most requests, but long coding or reasoning requests keep getting bumped. Earliest-deadline-first, from Liu and Layland's 1973 real-time work, has a different blind spot. Two requests with the same deadline look identical, even if one needs fifty milliseconds of work and the other needs twenty seconds.

10
00:02:38,756 --> 00:02:42,703
[Hal Turing] So the missing quantity is slack. Deadline minus the work still to do.

11
00:02:42,703 --> 00:03:00,118
[Dr. Ada Shannon] Right, that's close to least-laxity-first from real-time systems. Cascade calls it the latency budget: the class TTFT target minus an estimated TTFT for that request. Queueing, contention, preemption and KV transfer all spend from it, and it shrinks as the request waits.

12
00:03:00,118 --> 00:03:04,205
[Hal Turing] One more piece of mechanism. Why does the cache matter for scheduling at all?

13
00:03:04,205 --> 00:03:35,459
[Dr. Ada Shannon] Prefix caching. If a new request shares a prefix with earlier work, like a system prompt, a previous chat turn or a code repository, you reuse that state and skip prefill compute. But the cache is tiered. GPU memory, HBM, is fastest. CPU DRAM is the warm tier, reached over NVLink-C2C on GB200. NVMe is the deep tier, reached through GPU-Direct Storage. A deep-tier hit saves compute but costs transfer time.

14
00:03:35,459 --> 00:03:37,363
[Hal Turing] Does the transfer time actually hurt?

15
00:03:37,363 --> 00:04:00,815
[Dr. Ada Shannon] The paper's Figure 4 says yes. A 64,000-token prefix fetched from NVMe on Qwen-2.5-72B blows the TTFT target, even with LMCache's batched prefetch. Existing caches decide what to keep by capacity, reuse probability and transfer cost. They don't ask whether the requesting user can afford the wait.

16
00:04:00,815 --> 00:04:02,812
[Hal Turing] There's a third ingredient, chunked prefill.

17
00:04:02,812 --> 00:04:22,781
[Dr. Ada Shannon] From Sarathi-Serve, Agrawal and colleagues at Microsoft Research India, OSDI 2024. The engine splits prefill into fixed chunks, 512 tokens here, and mixes them with decode steps. Decodes aren't paused for a long prompt. Cascade leans on that to protect TPOT, which matters later.

18
00:04:22,781 --> 00:04:25,614
[Hal Turing] The paper opens with three motivation findings. What are they?

19
00:04:25,614 --> 00:04:47,766
[Dr. Ada Shannon] First, aggregate SLO attainment hides starved classes. A healthy-looking average can contain a class that's failing. Second, deep-tier restores need admission control, as Figure 4 showed. Third, violations come mostly from queueing, not from a lack of intrinsic budget. Unblocked, requests had roughly two to over twenty seconds of TTFT headroom across the three models.

20
00:04:47,766 --> 00:04:52,178
[Hal Turing] That third one is the hopeful part. The time exists. The queue wastes it.

21
00:04:52,178 --> 00:05:12,565
[Dr. Ada Shannon] Yes. And it's why fairness shows up as a metric. Jain's fairness index, from Jain, Chiu and Hawe at DEC in 1984, runs from one over n to one. The paper splits requests into six input-length bins and computes SLO attainment in each. One means every bin attains at the same rate. One-sixth is the worst case.

22
00:05:12,565 --> 00:05:14,794
[Hal Turing] Equal attainment, though, not equal service.

23
00:05:14,794 --> 00:05:20,599
[Dr. Ada Shannon] Exactly, so it rewards uniform failure as much as uniform success. You read it next to goodput.

24
00:05:20,599 --> 00:05:22,875
[Hal Turing] Anything to flag before we get to the design?

25
00:05:22,875 --> 00:05:37,457
[Dr. Ada Shannon] One thing. The headline cluster results come from an extended Vidur simulator, fed with data-movement profiles measured on GB200 NVL72. We'll hold the question of what ran on real hardware until we've seen what the system does.

26
00:05:37,457 --> 00:05:41,404
[Hal Turing] Let's walk through the design. A request arrives. What happens to it first?

27
00:05:41,404 --> 00:06:10,104
[Dr. Ada Shannon] An estimator predicts its time to first token, using the prompt length, how much of the prefix is cached at each tier, and the instance's current load. It runs the prediction twice, once at the actual load and once at the maximum load the batch could reach. It averages the two and multiplies by a guardband, gamma, that is at least one. The paper sets gamma once per model, offline, and holds it fixed across all traces. Budget is the class TTFT target minus that estimate.

28
00:06:10,104 --> 00:06:13,448
[Hal Turing] Why average with the worst case instead of trusting the actual load?

29
00:06:13,448 --> 00:06:26,683
[Dr. Ada Shannon] An optimistic estimate inflates the budget, and then the scheduler happily spends delay the request can't afford. Averaging with the worst case plus a margin leans conservative. The cost is that some requests look more doomed than they are.

30
00:06:26,683 --> 00:06:27,844
[Hal Turing] And the doomed ones?

31
00:06:27,844 --> 00:06:46,838
[Dr. Ada Shannon] A positive budget goes to a Tier-1 queue. A negative budget goes to Tier-2, which is best-effort. Those requests aren't dropped. A fixed fraction of every batch is reserved for Tier-2 so it can't starve. When load falls and a request's budget turns positive again, it gets promoted back. The paper doesn't say what that fraction is.

32
00:06:46,838 --> 00:06:51,621
[Hal Turing] Then ordering inside Tier-1. Where does it differ from earliest-deadline-first?

33
00:06:51,621 --> 00:07:14,377
[Dr. Ada Shannon] Remaining budget is the original budget minus time already waited, recomputed every chunk iteration. The queue is served least-remaining-budget-first. Two requests with the same deadline now separate, because the one needing more work has less budget left. And a long prompt with a loose SLO still has positive budget, so it isn't pushed to the back the way shortest-job-first pushes it.

34
00:07:14,377 --> 00:07:17,024
[Hal Turing] Now the cache side. How does the same budget gate a restore?

35
00:07:17,024 --> 00:07:42,194
[Dr. Ada Shannon] Restore time over a multi-hop path is the sum of each link's fixed overhead, plus the bytes times the sum of the inverse bandwidths. NVMe to DRAM to HBM is two hops. Invert that formula and you get the most bytes you can restore inside the remaining budget. Cascade restores up to that bound and recomputes the rest of the prefix on the GPU. So a deep-tier hit becomes partial, not all or nothing.

36
00:07:42,194 --> 00:07:46,281
[Hal Turing] So a 64k prefix on NVMe might be half restored and half recomputed.

37
00:07:46,281 --> 00:08:03,046
[Dr. Ada Shannon] Right, depending on what budget is left. HBM is allocated only at the final DRAM-to-HBM hop, so scarce memory isn't pinned during the slow leg. A cluster-wide index records the deepest tier holding each block, and eviction is plain LRU within each tier.

38
00:08:03,046 --> 00:08:05,275
[Hal Turing] What about preemption when HBM fills?

39
00:08:05,275 --> 00:08:22,086
[Dr. Ada Shannon] It evicts the prefill-stage request with the largest remaining budget, at a chunk boundary, and never a decode-stage request. The logic is that the request with the most slack can best absorb a redo. Its KV state spills to DRAM if that fits its budget, otherwise it's recomputed.

40
00:08:22,086 --> 00:08:26,684
[Hal Turing] Everything so far is about first-token time. Where does per-token latency come in?

41
00:08:26,684 --> 00:08:42,613
[Dr. Ada Shannon] It doesn't, directly. The budget bounds TTFT only. TPOT is delegated to chunked prefill, at 512-token chunks and a max batch of 128, so decodes keep moving. Cascade doesn't model TPOT itself.

42
00:08:42,613 --> 00:08:43,634
[Hal Turing] Now the setup.

43
00:08:43,634 --> 00:09:31,467
[Dr. Ada Shannon] Qwen-2.5-72B, Llama-3-70B and Llama-3-405B, tensor parallel 4, NVFP4 weights, FP8 KV cache, on GB200 NVL72. Qwen runs under multi-head attention, which stores more KV per token than its native grouped-query form. The workload is two-hour production traces from Aliyun Bailian in four classes, mixed into ten traces. SLOs are 10 times the isolated TTFT and 5 times the isolated TPOT, at p90. Baselines are FCFS, EDF and SJF behind the same global router, on clusters of 9, 6 and 12 instances. The cluster results run in an extended Vidur simulator fed with data-movement profiles measured on real GB200 hardware.

44
00:09:31,467 --> 00:09:32,721
[Hal Turing] So what do they find?

45
00:09:32,721 --> 00:10:04,440
[Dr. Ada Shannon] Averaged over ten traces and three models, Cascade delivers 2.4 times the goodput of vLLM's FCFS, with 40% fewer violations. It has the highest goodput on every trace. But the gain follows load. On the heavy mixed trace it's about two and three-quarters times. On the light traces it shrinks to between one-tenth and one-half more, and the paper credits that to prefetching, not reordering. Also, the text doesn't say whether 40% is relative or absolute, and there are no error bars.

46
00:10:04,440 --> 00:10:06,112
[Hal Turing] Where do the violations come from?

47
00:10:06,112 --> 00:10:29,192
[Dr. Ada Shannon] The paper sorts them by cause. On the Llama models, head-of-line blocking dominates. On Qwen, restoration and preemption weigh more, because multi-head attention means bigger KV per token. One Qwen example drops from 150,000 violations to 4,000. That's an illustration. It's hard to square with a 40% average, and the text doesn't reconcile them.

48
00:10:29,192 --> 00:10:34,440
[Hal Turing] The worry with any fast scheduler is that it buys speed by sacrificing the long requests. Does it?

49
00:10:34,440 --> 00:11:06,576
[Dr. Ada Shannon] On the mixed trace, no. SJF fails 12 to 16% of the shortest requests but 65 to 72% of the longest 1%, so about two in three. Cascade fails between 2 and 15% in every length bin and every model, so even the longest requests fail at most about once in seven. The fairness index tells the same story: Cascade sits between 0.98 and 1.0 on every trace, while SJF falls to about 0.6 on two Llama-405B traces.

50
00:11:06,576 --> 00:11:09,548
[Hal Turing] FCFS scores 0.95 or better there too, though.

51
00:11:09,548 --> 00:11:22,319
[Dr. Ada Shannon] Yes, and that matters. FCFS and EDF fail 50 to 70% of requests in every bin, and still score well, because the index measures equal attainment, not good attainment. So read it next to goodput.

52
00:11:22,319 --> 00:11:23,666
[Hal Turing] Last piece, the tail.

53
00:11:23,666 --> 00:11:42,428
[Dr. Ada Shannon] FCFS and EDF show two TTFT peaks, the second at 100 to 1,000 seconds, from requests stuck behind a long prompt. SJF removes that peak for short classes but moves it onto Coder and Reasoning. Cascade has a single peak near a tenth of a second in all four classes.

54
00:11:42,428 --> 00:11:49,394
[Hal Turing] Time for the skeptic's questions. The headline numbers come from a simulator. What was actually run on real hardware?

55
00:11:49,394 --> 00:12:17,908
[Dr. Ada Shannon] The paper says it implements Cascade on vLLM with LMCache, and I take that at face value. But every policy comparison is described as running under identical cluster settings inside an extended Vidur simulator. The real GB200 NVL72 contributes data-movement profiles. The text shows no end-to-end vLLM plus LMCache run under load, no measured goodput or violation counts, and no validation of the simulator against a live cluster.

56
00:12:17,908 --> 00:12:22,877
[Hal Turing] But the estimator is validated. Figure 7 compares predicted and actual TTFT.

57
00:12:22,877 --> 00:12:47,072
[Dr. Ada Shannon] On Qwen-2.5-72B, with a mean absolute error and a correlation. The values aren't in the text I have. The problem is that the estimator and the simulator are both calibrated through Vidur-style profiling. If they share modeling assumptions, they can agree with each other and both be wrong about a real cluster. What would settle it is a live trace replay with measured TTFT.

58
00:12:47,072 --> 00:12:54,549
[Hal Turing] Now the sweeps. When the capacity and load sweeps got pushed, FCFS collapsed. Is that a fair comparison?

59
00:12:54,549 --> 00:13:24,781
[Dr. Ada Shannon] Part of it is robust. At 48 queries per second, Cascade's p95 first-token time stays around 3.5 seconds, under its SLO, while FCFS and EDF cross the SLO by 36 queries per second. And Cascade beats SJF, the best baseline there, by about 3.2 times. Cascade also reaches 2.4 times FCFS's full-capacity goodput on 7 Qwen instances, 28 GPUs, where FCFS uses 9, 36 GPUs.

60
00:13:24,781 --> 00:13:25,617
[Hal Turing] And the rest?

61
00:13:25,617 --> 00:13:58,265
[Dr. Ada Shannon] At 48 QPS, FCFS and EDF deliver 0.05 times FCFS's goodput at 32 QPS, with over 90 percent violations. Cascade's 1.5 times is a thirty-fold gap. The baseline has stopped delivering, which isn't the same as Cascade being thirty times smarter. At 32 QPS all four policies are close, and the light traces give only 1.1 to 1.5 times. So the benefit is regime-driven. The 22 percent GPU saving is one model, one trace, in simulation, not a fleet measurement.

62
00:13:58,265 --> 00:14:00,169
[Hal Turing] What comparisons would you want instead?

63
00:14:00,169 --> 00:14:24,225
[Dr. Ada Shannon] Tuned baselines, plus admission control or load shedding, since an overloaded operator would use them. Mooncake, from Qin and colleagues in 2025, rejects requests under overload, and Cascade positions itself against that. Demoting negative-budget requests to a best-effort tier is a softer relative of rejection. But the paper doesn't report what fraction lands in Tier-2 or what happens to those requests.

64
00:14:24,225 --> 00:14:26,454
[Hal Turing] The closest prior art is QoServe, right?

65
00:14:26,454 --> 00:14:49,023
[Dr. Ada Shannon] QoServe, from Goel and colleagues at ASPLOS 2026, uses per-request slack and multiple SLO classes to resize chunks. Cascade discusses it, says it lacks prefix-cache reuse, and doesn't evaluate it. That's plausible, but it leaves open how much of Cascade's gain a slack-aware chunking scheme would capture on the same traces.

66
00:14:49,023 --> 00:14:53,017
[Hal Turing] What about the claim that one budget drives both scheduling and the cache?

67
00:14:53,017 --> 00:15:14,798
[Dr. Ada Shannon] Untested in isolation. There's no ablation with budget scheduling but ordinary cache handling, or the reverse. The paper attributes the light-load gains to prefetching and the heavy-load gains to reordering, but no controlled run separates them. My read is that the restore-or-recompute rule is the more novel half, since ordering by slack is close to least-laxity-first.

68
00:15:14,798 --> 00:15:16,562
[Hal Turing] Fairness was the other big claim.

69
00:15:16,562 --> 00:15:42,104
[Dr. Ada Shannon] Cascade defines fairness as equal SLO attainment across length bins. The Virtual Token Counter work, "Fairness in Serving Large Language Models" by Sheng and colleagues at Berkeley in 2024, defines it as equal token service per tenant. Cascade neither cites nor compares against it, and per-tenant fairness isn't tested. So Cascade's real fairness evidence is the flat violation rate across bins, not the index.

70
00:15:42,104 --> 00:15:44,240
[Hal Turing] And first-token time versus per-token time?

71
00:15:44,240 --> 00:16:04,720
[Dr. Ada Shannon] The budget is TTFT-only. Per-token latency is delegated to chunked prefill, with a fixed 512-token chunk. Chunk size and attention cost growing with context bound what that protects. No TPOT-specific results are reported, so the goodput gain can't be split into first-token and per-token effects.

72
00:16:04,720 --> 00:16:05,835
[Hal Turing] Any smaller gaps?

73
00:16:05,835 --> 00:16:34,674
[Dr. Ada Shannon] A few. The guardband is fixed per model and its sensitivity isn't reported. The output-length prediction the intro mentions isn't described. There are no confidence intervals. It's one two-hour trace family with synthetic mixes. The hardware is GB200 with fast links, so commodity PCIe clusters are untested. And Qwen runs under multi-head attention rather than its native grouped-query form, which inflates cache size and shapes where its violations come from.

74
00:16:34,674 --> 00:16:36,810
[Hal Turing] So what should a practitioner take from this?

75
00:16:36,810 --> 00:16:55,294
[Dr. Ada Shannon] The contribution is one explicitly tracked budget that unifies ordering, restore-or-recompute and victim selection, with no new hardware or weight changes. The design rule that a deep-tier cache hit is optional, not automatically good, stands on its own given the 64k NVMe example.

76
00:16:55,294 --> 00:16:56,640
[Hal Turing] And what would you want next?

77
00:16:56,640 --> 00:17:12,383
[Dr. Ada Shannon] A live vLLM plus LMCache replay to validate the simulator, tuned or admission-controlled baselines, QoServe and VTC comparisons, per-tenant fairness, disaggregated prefill and decode, and ablations of the estimator and guardband.

78
00:17:12,383 --> 00:17:38,808
[Hal Turing] So in a calibrated simulator, on production-derived traces, budget-ordered scheduling with budget-gated restores beats FCFS, EDF and SJF, especially near saturation, without SJF's penalty on long requests. The boundary is that there's no live-cluster evidence, the baselines collapse in the headline regimes, and fairness means attainment parity. Thanks for listening, everyone.

79
00:17:38,808 --> 00:17:39,597
[Dr. Ada Shannon] Thanks, all.