TreeWY: Speculative Verification for Gated DeltaNet Hybrids

This episode examines TreeWY, a proposed method for speculative decoding verification in hybrid language models that mix standard attention with Gated DeltaNet linear-attention layers. The discussion explains why current systems like vLLM and SGLang must snapshot the full recurrent state at every draft position before verification, since GDN's decay-and-overwrite state update can't be partially rolled back — a cost that multiplies across branches and makes wide speculative draft trees prohibitively memory-expensive. It traces the problem to its root, from the memory-bandwidth bottleneck that motivates speculative decoding in the first place to the mathematical mechanics of the gated delta rule that make hybrid-model states lossy and irreversible. The paper's proposed fix reframes the state update as decayed additive attention with a corrected value, hinting at a way to verify an entire draft tree with a single triangular solve rather than exhaustive snapshotting. Listeners interested in LLM inference efficiency will find the episode's central claim striking: a roughly 128x reduction in per-node memory without sacrificing correctness guarantees, potentially unlocking much more aggressive tree-based speculation on hybrid architectures.