98× Faster LLM Routing with Flash Attention for vLLM Semantic Router

Where the router sits

Step through the three stages. Each adds one change to the pipeline in front of the LLM (8K-token prompt).

Mask memory vs the budget Log scale

The ONNX export builds an n×n mask per session. Per the paper: ~1.5 GB at 8K, ×4 per doubling. Budget line: ~718 MB.

Standard vs tiled attention Illustrative: 16×16 toy matrix

Standard SDPA stores every score. FlashAttention visits tiles, keeps a running softmax, and discards each tile. Result is exact.

ModernBERT layer pattern Illustrative: small window

Real kernel arguments: local layers use window 63 left / 64 right; global layers use −1 / −1. Click a layer chip; every third layer is global.

Sentence selection to ~512 tokens Illustrative: synthetic sentence scores

Four signals, paper defaults 0.20 / 0.40 / 0.35 / 0.05. First 3 and last 2 sentences always kept. Drag weights to see what is kept.

What compression did (reported)

The ladder at 8K tokens

Left: paper's measured ladder. Right option: the hosts' split using the paper's estimated ~0.8 s CPU+compression row.

FlashAttention vs plain attention by length 32K = derived/estimated

Body handling: three paths

Pick an Envoy path. Boxes in the top lane are body chunks.

Cleanest streaming evidence

Domain-only, 16K tokens, end-to-end. The 62 → 50 ms ladder step bundles streaming with the JSON change.

How firm is each claim? Hover cells