A decode-first view of LLM hardware: not who wins the biggest batch benchmark, but who makes the next token appear fastest. This page turns the episode into diagrams, heatmaps, scaling plots, and claim-stress tests centered on latency, memory traffic, and synchronization.
LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference
Main source paper. arXiv:2408.07326
DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation
Contrast case for low-latency generation on FPGA-based systems. Scholar
SpAtten
Sparse attention and pruning as a “reduce the traffic” alternative to a denser fast path. Scholar
PagedAttention
Software-side serving efficiency and KV memory management pressure. Scholar
DistServe
Phase split serving perspective for prefill versus decode. Scholar
TokenWeave and communication studies
Later work on overlap and distributed inference communication patterns. TokenWeave · Communication patterns
Additional arXiv IDs extracted from the transcript with pattern DDDD.DDDDD: none beyond 2408.07326.
Listen to the Episode
Interactive hardware only matters if it changes the pause a human notices between tokens.