Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving

One budget, three jobs

Budget = class TTFT target − estimated TTFT. It orders the queue, gates cache restores and picks preemption victims. Click a stage.

Three motivating findings

Queue Lab Illustrative toy

Nine requests all waiting at t=0 on one non-preemptive server. Numbers are invented to show the mechanisms, not paper data.

Budget spent while waiting Illustrative toy

Each cell is the fraction of a request's initial budget already consumed at time t. Hot rows run out first. J9 starts negative, so it goes to Tier-2.

Latency budget estimator Illustrative inputs

Estimate = γ × average(TTFT at actual load, TTFT at max batch load). Budget = target − estimate. Remaining = budget − time waited.

Restore or recompute Illustrative inputs

Restore time T = Σ oᵢ + B · Σ (1/BWᵢ). Invert for B: the most bytes restorable inside the remaining budget. Recompute the rest.

Reported results

All cluster numbers come from an extended Vidur simulator fed with GB200 NVL72 data-movement profiles.

What ran where

Evidence ledger

Hover a row. Based on what the episode says the text reports.