Workload–Router–Pool: Unifying LLM Routing, Models, and GPU Fleets

Chen, Liu, He, Lyu, Chen, Liu, Liu, Jiang · vLLM Semantic Router project · arXiv Mar 2026 (v2 Apr)
arXiv 2603.21354 Vision paper · no new experiments Every number is inherited

The 3×3 interaction matrix

Rows drive, columns are driven. Hover or tap a cell.

Select a cell.

Short chat in a 64K-context slot

Slider changes request length. Cells are 1K tokens each (arithmetic on the paper's 64K config).

Request:8K

Agent session: tool pauses and cache eviction

IllustrativeWidths are schematic, not measured. The paper cites 40–60% of an agent session as paused tool calls.

Cheap signals → workload prior

Toggle the request type; hover a signal.

Hover a signal.

Feedback-driven adaptation

Three-tier vision-language cascade (AVR)

ProjectedThe 78% figure is a projection, not a measurement.

RouteLLM's cited 95% of GPT-4 quality at 26% cost uses a different quality definition; the two percentages should not be lined up.

FleetOpt: choosing the pool boundary B*

IllustrativePrompt-length histogram is a schematic long-tail shape, not paper data. Measure your own distribution.

Boundary:8K

The 1/W law: H100 energy efficiency vs context

ReportedAnalytical model values from the paper.

Router latency vs first-token budget

Splitwise and DistServe report 4.48× more requests at equal latency targets. The authors' own simulator (A10G, A100, H100) finds KV transfer can outweigh disaggregation for short prompts at a small boundary, and an A10G fleet can beat H100s when concurrency matters more than per-token speed. Simulator outputs.

Provenance audit: how was each number produced?

Cell intensity = how strongly the transcript ties the number to that source. Click a row.

Click a row.

Thesis 1: how big is the measured coupling gain?

ReportedRanges as stated in the paper.

Hover a bar.

What would upgrade vision to result

References

  1. The Workload-Router-Pool Architecture for LLM Inference Optimization — Chen et al., 2026 (arXiv 2603.21354)
  2. FrugalGPT — Chen, Zaharia, Zou, 2023
  3. RouteLLM — Ong et al., 2024
  4. Hybrid LLM — Ding et al., 2024
  5. RouterBench — Hu et al., 2024
  6. Splitwise — Patel et al., 2024
  7. DistServe — Zhong et al., 2024
  8. Vidur — Agrawal et al., 2024
  9. Mélange — Griggs et al., 2024
  10. BurstGPT — Wang et al., 2024
  11. Mooncake — Qin et al., 2024
  12. LMSYS-Chat-1M — Zheng et al., 2023
  13. vLLM / PagedAttention — Kwon et al., 2023
  14. GPT Semantic Cache — Regmi, Pun, 2024
  15. MeanCache — Gill et al., 2024
  16. AutoMix — Aggarwal et al., 2023
  17. ServeGen — Wang et al., 2025
  18. Router-R1 — Zhang et al., 2025
  19. Preble — Srivatsa et al., 2024