1
00:00:01,000 --> 00:01:00,024
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

Today we're on You Only Cache Once: Decoder-Decoder Architectures for Language Models, by Yutao Sun and eight co-authors, from Microsoft Research and Tsinghua University, posted to arXiv in May 2024. The question is blunt: can you compute one global key-value cache, reuse it in every upper layer, keep decoder-only quality, and still cut long-context memory and prefill time by large factors? Here's the setup. A 65-billion-parameter model with grouped-query attention and 8-bit cache quantization needs about 86 gigabytes of cache at 512K tokens. That's more than a whole H100 has, before you count the weights. Meanwhile a 7B model needs about 110 seconds to prefill 450K tokens on four H100s.

2
00:01:00,024 --> 00:01:27,174
[Dr. Ada Shannon] So a model can be perfectly capable of reading a million tokens and still be undeployable, because its scratch notes are bigger than the GPU, and you wait minutes for the first word. YOCO claims about an 80 times smaller cache at 65B, 71.8 times faster prefill at a million tokens, and 9.6 times the throughput at 512K, with quality on par with a Transformer. Those are enormous multipliers. My first question is what they're multiplied against, and we'll get to that.

3
00:01:27,174 --> 00:01:43,699
[Hal Turing] Hold that thought, because I want to understand the object being shrunk first. I've said the phrase KV cache a hundred times, but if you asked me what's stored and why the size hurts so much, I'd hand-wave. Walk me through it like I've never tuned a serving stack.

4
00:01:43,699 --> 00:02:27,849
[Dr. Ada Shannon] Every attention layer keeps a key vector and a value vector for every past token, so generation never recomputes history. The cost is that it grows with layers times tokens times hidden size, and at long context it competes with the weights for GPU memory. Two phases matter. Prefill encodes the whole prompt in parallel, its attention cost is quadratic in prompt length, and it sets time to first token. Decode then emits one token at a time, and every step reads the entire cache, so it's limited by memory bandwidth, not compute. The existing levers are grouped-query attention, from Ainslie and colleagues at Google Research in 2023, where many query heads share a smaller set of key-value heads, and quantization, which means fewer bits per element. The paper's own numbers already use both. The 86 gigabytes is what's left after the easy wins.

5
00:02:27,849 --> 00:02:44,724
[Hal Turing] The intro sorts language models into three layouts: BERT-style encoder-only, T5-style encoder-decoder, and GPT-style decoder-only. I know decoder-only won, but the only reason I can give is that everyone did it. What's the crisp version?

6
00:02:44,724 --> 00:03:16,599
[Dr. Ada Shannon] Autoregressive generation. Encoder-only models read bidirectionally, so every new token means re-encoding the input. Encoder-decoder models, from the original Attention Is All You Need paper by Vaswani at Google in 2017 through T5 by Raffel at Google in 2020, feed a bidirectional encoder into a causal decoder through cross-attention. In multi-turn chat that growing history keeps needing re-encoding. Decoder-only models just cache keys and values and take one incremental step per token: one stack, simple training, cheap generation.

7
00:03:16,599 --> 00:03:30,474
[Hal Turing] Wait wait wait— cross-attention from a decoder into a separate stack's output? That's exactly what this paper does, isn't it? So how is it not an encoder-decoder, with the re-encoding problem you just described?

8
00:03:30,474 --> 00:04:15,625
[Dr. Ada Shannon] It borrows the wiring and drops the bidirectionality. Split the L layers in half. The bottom half is the self-decoder, built from efficient self-attention, meaning any attention variant whose state doesn't grow with length. Here that's sliding-window attention or gated retention, a recurrent-style layer we'll unpack later. Its output is projected once into a single set of global keys and values. The top half is the cross-decoder: each layer has its own queries and cross-attends to that one shared cache. Everything stays causal, so from outside it takes a prefix and predicts the next token like any decoder-only model. The cache shrinks from L layers' worth to roughly one, an L-fold saving. One footnote: the self-decoder still holds constant-size state, so "once" means the global cache. And since the cross-decoder's keys and values come only from the self-decoder, prefill can skip the cross-decoder for the prompt tokens.

9
00:04:15,625 --> 00:04:29,500
[Hal Turing] I'll give them this: the saving falls out of the wiring, not out of a tuned heuristic that might misfire, and that's a clean claim. But every upper layer now reads the same memory. Somebody has to pay for that.

10
00:04:29,500 --> 00:04:48,375
[Dr. Ada Shannon] Somebody does. Each cross-decoder layer has less freedom over what it stores, and the paper's scaling results are meant to show that costs little. Next we cover how the mechanism works and what they measured. After that, whether the comparisons behind those big multipliers hold up, because that's where the interesting argument lives.

11
00:04:48,375 --> 00:05:00,125
[Hal Turing] Okay, the wiring is in my head. Now show me the plumbing. What does each half compute, and where does the saving show up on paper? Consequence first, if Ada's allowed to do that.

12
00:05:00,125 --> 00:05:40,850
[Dr. Ada Shannon] Consequence first: cache memory goes from order L times N times D to order N plus L, times D. Prefill attention time goes from N squared to linear in N. Here's the mechanism. Self-decoder layers run efficient attention on normalized input plus a residual, then a SwiGLU feed-forward. At the halfway point the output is normalized once and projected by a single pair of matrices into the shared keys and values. Each cross-decoder layer keeps only its own query projection, attends causally to that one pair, and works with grouped-query heads. Prefill gets two discounts: half the layers can be skipped, and the self-decoder is linear-time. The skip is legal because nothing in the cross-decoder ever feeds back into the—

13
00:05:40,850 --> 00:05:51,625
[Hal Turing] Sorry to cut you off, but the last prompt token still has to produce the first output token. Doesn't something have to run the top half, or you'd have no logits?

14
00:05:51,625 --> 00:06:22,100
[Dr. Ada Shannon] Right, that's the nuance. In Figure 3, prefill runs the self-decoder over the whole prompt, writes the shared KV, and skips the cross-decoder for the prompt tokens. Only the final position climbs through the cross-decoder. The output is exactly unchanged, which is unusual for a speedup. Training gets a matching trick, chunk parallelism, for 1M-token sequences. The self-decoder only swaps state with neighboring devices, and the cross-decoder does one all-gather of KV instead of one per layer.

15
00:06:22,100 --> 00:06:29,675
[Hal Turing] Sliding window I follow: fixed window, constant cache. Gated retention is the piece I don't have yet.

16
00:06:29,675 --> 00:07:03,824
[Dr. Ada Shannon] It's retention, from the Retentive Network paper out of Microsoft Research in 2023, plus a data-dependent decay gate. The gate is a sigmoid of the input raised to one over a temperature, and the temperature pushes decay toward one so the model remembers longer. It's a head-wise scalar, not per-element, so it maps onto tensor cores. It has three equivalent forms. Parallel is for training. Recurrent is for decoding, with one d-by-d state per head and constant memory. Chunkwise is for prefill at chunk size 256: parallel inside a chunk, recurrent across chunks. They wrote a Triton kernel on the FLA library.

17
00:07:03,824 --> 00:07:09,224
[Hal Turing] Good. Now the measurements. What did the 3B model score, and against whom?

18
00:07:09,224 --> 00:08:18,399
[Dr. Ada Shannon] YOCO-3B has 26 layers, hidden size 3072, 24 query and 8 KV heads, and trained on 1.6 trillion tokens with the StableLM-3B-4E1T recipe. Average zero-shot accuracy is 0.634 at 1T tokens and 0.636 at 1.6T. OpenLLaMA-3B-v2, from Berkeley, gets 0.619 at 1T, and Stability AI's StableLM-alpha-3B-v2 gets 0.612. Those models used different data and tokenizers, so treat it as a reference point. The same-data test is the scaling study: 160M to 13B parameters, 10B tokens, 2k context. YOCO-gRet lands at or below Transformer loss, and sliding-window is about equal. At 160M it's 3.530 versus 3.564, with 3.553 for sliding-window. Associative-recall loss is 1.199 versus 1.219. Mamba is 3.645 and RetNet 3.633. The authors credit complementary attention and retention biases.

19
00:08:18,399 --> 00:08:23,125
[Hal Turing] And the million-token claim? How did they get there, and how did it do?

20
00:08:23,125 --> 00:09:22,625
[Dr. Ada Shannon] They continued training the 3B on a 64K, 256K, then 1M schedule, with 6B, 4B and 1.5B tokens, RoPE theta 640K, 5M, then 80M, and learning rates falling from 8e-5 to 2e-5. Needle-in-a-haystack is near-perfect out to 1M, ten runs per cell. Cumulative loss keeps falling with length on books and repo-level code. Multi-needle at 128K, for one, two, four and eight needles: YOCO scores 0.98, 0.98, 0.84, 0.56. LWM-1M-text, a 7B from Berkeley in 2024, scores 1.00, 0.90, 0.76, 0.62. MiniCPM-128K out of Tsinghua scores 1.00, 1.00, 0.54, 0.56. So YOCO wins at four needles, trails LWM at eight, ties MiniCPM there, and sits slightly behind at one. The Table 3 average reaches 0.645 after 11.5B more tokens.

21
00:09:22,625 --> 00:09:26,049
[Hal Turing] Alright, the numbers on the cover. Inference.

22
00:09:26,049 --> 00:10:53,049
[Dr. Ada Shannon] H100-80GB, lengths 32K to 1M, generating the last 1,024 tokens. The Transformer baseline uses GQA, Flash-Decoding from Tri Dao's group in 2023, and kernel fusion. Memory reduction is 1.95x at 32K, 3.01x at 128K, 6.39x at 512K and 9.38x at 1M, where YOCO totals 12.4 GB. Per-token KV savings track layer count, from 24x at 1.2B up to 80x at 65B. A 65B YOCO serves 128K tokens in 1 GB, versus 1.6K for GQA. Prefill is 2.87x faster at 32K, 8.36x at 128K, 30.3x at 512K (180 seconds to under 6), and 71.82x at 1M. Throughput runs 2.72x at 32K up to 9.56x at 512K, 4.5 versus 43.1 tokens per second, from cheaper prefill plus bigger batches. One flag: Section 4.4 quotes about 300 seconds for the 1M Transformer prefill, while the introduction says 380 for a 7B on four GPUs. Those are different setups.

23
00:10:53,049 --> 00:11:16,849
[Hal Turing] Ada, before I sharpen anything, a genuine question. The Transformer at 512K and 1M was never trained at that length, and only YOCO-3B-1M was extended. So what is the 71.8x actually measuring? I'll say one thing plainly first: the memory breakdown in Figure 7b is clean accounting, and the early exit is exact.

24
00:11:16,849 --> 00:12:31,799
[Dr. Ada Shannon] It measures a same-shaped Transformer run as a systems test, and nothing more. The 71.8x is the gap to a configuration few would ship. That configuration is full-attention GQA at a million tokens, with no KV quantization, no eviction, and no local layers. Real deployments use 4 or 8-bit KV, eviction, or local and global layer mixes like Mistral and Gemma. The paper's own motivating example assumes 8-bit KV, yet Figure 8 compares against an unquantized cache. It's a race where the opponent left the parking brake on, and the paper reports the lap time.

The missing opponents are known. Cross-Layer Attention, by William Brandon at MIT and colleagues in 2024, shares KV across adjacent layers for about 2x savings with small quality loss. Layer-Condensed KV Cache, by Haoyi Wu and Kewei Tu at ShanghaiTech in 2024, pairs every layer's queries with only the top layer's KV, at a training cost. Multi-head Latent Attention, from DeepSeek-V2 out of DeepSeek-AI in 2024, compresses KV into a low-rank latent. YOCO is the extreme end of that depth-sharing family, and the paper runs none of them. They're mostly orthogonal, so the real question is the marginal gain over a tuned stack.

25
00:12:31,799 --> 00:12:44,474
[Hal Turing] Hold on, the 80x bothers me too. Figure 8 is a 65B number, but the measured whole-system win at 3B and 1M is 9.4x. Where did the rest go?

26
00:12:44,474 --> 00:13:27,399
[Dr. Ada Shannon] Arithmetic versus measurement. The 80x is per-token KV at 65B, computed rather than run. At 3B the weights and the self-decoder state don't shrink. There's a subtler gap too, capacity versus bandwidth. Each cross-decoder layer still streams the full shared KV at every decode step, so per-token decode work isn't cut L-fold. The freed memory buys batch size, which is the mechanism in Reiner Pope's Efficiently Scaling Transformer Inference from Google in 2022. But 4.5 versus 43.1 tokens per second at 512K blends a 512K prefill with 1,024 generated tokens, and prefill dominates. There's no decode-latency-per-token plot. That's an open question, not a proven flaw.

27
00:13:27,399 --> 00:13:41,799
[Hal Turing] And quality? Table 3 has YOCO-3B at 0.634 to 0.636. Does an Eval Harness average have the resolution to catch what sharing one cache across half the layers costs?

28
00:13:41,799 --> 00:14:40,124
[Dr. Ada Shannon] No. The gap is 0.02 to 0.03 against models with different data and tokenizers, with no seeds or confidence intervals. The scaling curves reach 13B, but at 10B tokens and 2K context, roughly 0.8 tokens per parameter, far from compute-optimal. And the gRet win comes from a retention-attention hybrid. The paper itself says 1:3 interleaving gets similar gains. Jamba, from Opher Lieber and AI21 Labs in 2024, shrinks cache exactly that way, with attention in about one layer in eight. So is the gain from cache-once wiring or from hybridization? You need an interleaved hybrid at matched parameters.

Long context has the same problem. A city with a magic number is the easiest needle. RULER, from Cheng-Ping Hsieh and NVIDIA in 2024, showed perfect needle scores can hide collapse on multi-hop and aggregation tasks. And cumulative NLL is dominated by early tokens.

29
00:14:40,124 --> 00:14:49,999
[Hal Turing] Then the capacity question. Thirteen upper layers all read one middle-layer KV. Does that survive going from 26 layers to 80?

30
00:14:49,999 --> 00:15:34,299
[Dr. Ada Shannon] Nobody knows. The 50:50 split is asserted, never ablated, and there's no per-layer attention entropy to show whether cross layers retrieve differently. One cache is also one point of failure. H2O from Zhenyu Zhang at UT Austin in 2023 and SnapKV from Yuhong Li at UIUC in 2024 exploit per-layer tolerance to eviction. Here a bad quantization or eviction hits all 13 layers at once, and the paper never tests it. Retrofitting is also unanswered. SwiftKV, from Aurick Qiao at Snowflake in 2024, gets a similar prefill skip on existing models through distillation, though approximately. YOCO needs pretraining from scratch. What would settle it is a matched 3B Transformer with the same 64K, 256K, 1M schedule, RULER at 1M, and an 80-layer run.

31
00:15:34,299 --> 00:15:37,774
[Hal Turing] So what holds up, and when would you reach for it?

32
00:15:37,774 --> 00:16:00,849
[Dr. Ada Shannon] The early-exit prefill is architectural and exact, and linear-time prefill is a real complexity gain. One shared cache is a clean hook for a single index, compression, or pre-cached context. It pays off for very long prompts, prefill-dominated workloads, and memory-limited serving. The advantage shrinks when prefixes are cached, since early exit then saves nothing, and retention state must be snapshotted per prefix rather than sliced like KV blocks.

33
00:16:00,849 --> 00:16:28,824
[Hal Turing] So my takeaway: YOCO is a genuine architectural idea with real, measured gains. The exact early exit and the linear prefill are solid. The headline multipliers, though, come from an unquantized full-attention baseline at lengths where nobody would run it, and the quality evidence is competitive rather than conclusive. How much survives a fair fight with MLA, CLA, and a tuned quantized stack is still open. Thanks for listening, everyone. See you next time.