BASED: Balancing Recall and Throughput in Linear Attention

This episode dissects the BASED architecture from Arora, Eyuboglu, Zhang et al. (Stanford/Buffalo), examining how it tries to resolve the tradeoff between recall accuracy and inference throughput in sequence models. The discussion traces the lineage of alternatives to standard softmax attention — state space models like Mamba, gated-convolution approaches like H3 and Hyena, linear attention, and sliding window attention — explaining why a growing KV-cache makes attention memory-bound at scale, while fixed-size-state models trade away precise recall by construction. It highlights the MQAR benchmark from the Zoology paper as a tool for exposing this recall-versus-state-size Pareto frontier, showing that every architecture, including full attention, sits somewhere on that curve rather than escaping it. The hosts then unpack BASED's hybrid design, which combines a linear attention component for cheap global context with a small sliding window for exact local comparisons, and start evaluating whether this combination actually delivers on its claimed 24x throughput gain over FlashAttention-2. Listeners interested in efficient LLM inference will get a clear framework for why architecture choices around memory and recall are fundamentally linked rather than independently solvable engineering problems.