1
00:00:01,000 --> 00:00:08,755
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,755 --> 00:00:35,040
[Hal Turing] Today's paper is "Learning What to Remember: Test-Time Training via Context Distillation," by Zixuan Wang, Xingyu Dang, and colleagues at Princeton, UC Berkeley, UC Santa Cruz, and Carnegie Mellon. It's an August 2026 arXiv preprint, and the code is released. It builds directly on In-Place Test-Time Training. The question is what a model should write into a fixed-size memory when it can't keep everything.

3
00:00:35,040 --> 00:00:52,176
[Dr. Ada Shannon] The claim is that you decide what to write by comparing two views of the same model: one that sees a long window and one that sees a short window. The gap between them says what the short view is missing. That gap gets written into the weights. Whether the gap really measures remote history is the tension we'll keep returning to.

4
00:00:52,176 --> 00:00:56,402
[Hal Turing] Start me at the beginning. Why does the model need a bounded memory at all?

5
00:00:56,402 --> 00:01:22,641
[Dr. Ada Shannon] Full attention keeps every past key and value in a cache. Lookup is nearly exact, but the cache grows with context, and compute grows quadratically. Linear-attention and recurrent models replace the cache with a fixed-size state updated per token, so cost per token is constant. DeltaNet is one, and Gated DeltaNet adds a gate that lets the state forget. The price is that the state is lossy, so the model has to decide what to keep.

6
00:01:22,641 --> 00:01:26,960
[Hal Turing] People like to compare that to human memory, where you decide what will matter later.

7
00:01:26,960 --> 00:01:39,220
[Dr. Ada Shannon] As motivation, fine. As evidence, no. A human doing that has goals. A fixed-capacity state only has whatever training objective you hand it. So the real question is which objective.

8
00:01:39,220 --> 00:01:41,588
[Hal Turing] And test-time training is where that objective lives?

9
00:01:41,588 --> 00:02:06,991
[Dr. Ada Shannon] Yes. Ordinary weights are frozen at inference. In test-time training, some weights, called fast weights, get gradient steps on the sequence while the model runs, so the weights themselves become the memory. The slow weights are the normal trained parameters, shared across sequences, and they learn how to use that memory. Think of a bounded notebook you keep rewriting as you read, instead of a growing pile of everything you've read.

10
00:02:06,991 --> 00:02:09,731
[Hal Turing] So DeltaNet is that notebook with a very simple notebook?

11
00:02:09,731 --> 00:02:33,740
[Dr. Ada Shannon] Right. The delta rule is one gradient step on a reconstruction loss with a single matrix as the state. The TTT-layer paper by Yu Sun and colleagues, 2024, generalized that so the state can be a small MLP. Titans goes further with a deeper memory. For speed, updates are applied per chunk of tokens rather than per token. The Large Chunk TTT paper, LaCT, does that.

12
00:02:33,740 --> 00:02:36,387
[Hal Turing] What's wrong with those objectives, according to the authors?

13
00:02:36,387 --> 00:03:04,669
[Dr. Ada Shannon] Their account has two parts. Reconstruction objectives, like original TTT, Titans, and LaCT, ask the state to reproduce stored key-value pairs. That spreads capacity evenly over everything, whether or not it will matter. Next-token objectives, like In-Place TTT and TTT-E2E, reward what helps predict the next token. The authors say that has a recency bias toward local signal. Those are their claims. We'll see how much of it they actually test.

14
00:03:04,669 --> 00:03:06,480
[Hal Turing] Where does In-Place TTT come in?

15
00:03:06,480 --> 00:03:34,437
[Dr. Ada Shannon] Feng and colleagues, ICLR 2026. Instead of adding a memory module, they reuse the MLP down-projection matrix of an ordinary transformer as the fast weight, updated chunk by chunk. The training target comes from a convolution over input embeddings, so it's aligned with language modeling. Because it adds no new architecture, you can retrofit a pretrained transformer. That's why it's the main baseline, and the base TTCD builds on.

16
00:03:34,437 --> 00:03:35,784
[Hal Turing] Now the distillation part.

17
00:03:35,784 --> 00:03:56,914
[Dr. Ada Shannon] Context distillation trains a student without some context to match a teacher that has it. Snell and colleagues did it in 2022, typically to bake a prompt into the weights. It's normally offline, a one-time training job. Cartridges and KV-Distill also compress context offline. TTCD does it online, per sequence, with no external teacher.

18
00:03:56,914 --> 00:03:59,097
[Hal Turing] No external teacher. Then who's the teacher?

19
00:03:59,097 --> 00:04:33,834
[Dr. Ada Shannon] The same frozen backbone, run twice. One pass uses sliding-window attention with a long window, and the other uses a short window. Sliding-window attention means each token only sees the last w tokens, and anything older is invisible unless some other state carries it. The long-window pass is the teacher and the short-window pass is the student. The difference between their MLP outputs is the residual: what the extra history contributed. TTCD writes that residual into the fast weights, so it survives after the evidence leaves the short window.

20
00:04:33,834 --> 00:04:35,599
[Hal Turing] That sounds clean. Where's the catch?

21
00:04:35,599 --> 00:05:07,735
[Dr. Ada Shannon] Two open questions. First, if the teacher window is only modestly longer than the student's, is that residual really remote history, or just a small window extension accumulated over time? Second, does the distillation loss actually shape the writes, or does the surrounding architecture do the work? We'll hold both until we've seen the method and the experiments. One more fact for later: the paper leaves TTT-E2E out of its comparisons, even though it's the closest next-token competitor. First, though, we need the actual update rule.

22
00:05:07,735 --> 00:05:11,636
[Hal Turing] Let's start with the loss. What does the student actually get penalized for?

23
00:05:11,636 --> 00:05:40,243
[Dr. Ada Shannon] The student's MLP output is pulled toward the teacher's MLP output, with the teacher detached. That's Equation 4, half a squared error. Take its negative gradient with respect to the down-projection matrix and you get the residual, student minus teacher, times the transpose of the student's own MLP activation. That's Equation 6. The residual says what to write. The student activation is the write key, so it decides which later tokens can read that write back.

24
00:05:40,243 --> 00:05:45,769
[Hal Turing] So it's an outer-product write, like the delta-rule models, but the value comes from the teacher gap.

25
00:05:45,769 --> 00:05:49,624
[Dr. Ada Shannon] Yes. The residual is the value and the activation is the address.

26
00:05:49,624 --> 00:05:51,017
[Hal Turing] What does the theory add?

27
00:05:51,017 --> 00:06:28,587
[Dr. Ada Shannon] Proposition 1 is a one-step existence argument in a single final-layer MLP. Suppose the supporting context sits inside the teacher window but outside the student window. Suppose the teacher has a real signal at an earlier query that the student lacks. Suppose a later query's student activation aligns with the earlier one, and is orthogonal to every other written position. Then one update moves the later student output toward the teacher's output by at least the learning rate times the gap times the alignment. It shows the mechanism can work. It says nothing about training at scale or about messy, non-orthogonal activations.

28
00:06:28,587 --> 00:06:31,513
[Hal Turing] And what actually runs? I'm guessing it isn't Equation 6.

29
00:06:31,513 --> 00:07:02,953
[Dr. Ada Shannon] The paper calls Equation 4 the loss and Equation 12 the practical update. In Equation 12 the residual passes through short convolutions and a learned projection, the keys are L2-normalized student activations, and the write is scaled by a learned coefficient. Activations are computed with the base down-projection, not the evolving one, so a causal parallel scan can accumulate the writes. That's the off-policy approximation. The outer training objective is still ordinary language modeling. I'll hold the judgment on that for the end.

30
00:07:02,953 --> 00:07:05,600
[Hal Turing] Give me the settings, since the critique will depend on them.

31
00:07:05,600 --> 00:07:38,200
[Dr. Ada Shannon] From scratch at 340M and 760M, the teacher window is 2048 and the student window is 1024. Training sequences are 32K, there are 4 TTT layers, and token budgets are 20 and 40 billion, with Muon. For the small synthetic MAD tasks, teacher and student are 32 and 16, or 16 and 8. In continual pre-training the student is 2048 and the teacher is full attention on 64K sequences. No sweep over the teacher-student ratio is reported.

32
00:07:38,200 --> 00:07:39,547
[Hal Turing] Who are the opponents?

33
00:07:39,547 --> 00:08:10,476
[Dr. Ada Shannon] From scratch: sliding-window attention, DeltaNet, Gated DeltaNet, and In-Place TTT. That baseline uses the input-embedding target, because the hidden-state target was unstable. For retrofitting, SmolLM2 360M and 1.7B and Llama-3.1-8B, against YaRN, attention-only continual training, and In-Place TTT. TTT-E2E isn't compared, and LaCT appears only on the synthetic suite.

34
00:08:10,476 --> 00:08:12,520
[Hal Turing] Retrieval first. What's the needle test?

35
00:08:12,520 --> 00:08:36,343
[Dr. Ada Shannon] RULER's needle-in-a-haystack hides a fact in long filler and asks for it back. Higher is better. Six sub-tasks, 100 examples each. At 760M, IP-TTCD has the best average at every length, and the gap over In-Place TTT widens. At 32K it averages about 22 correct per hundred, against about 9. That's roughly 2.4 times as many.

36
00:08:36,343 --> 00:08:37,597
[Hal Turing] Against the recurrent models?

37
00:08:37,597 --> 00:09:09,966
[Dr. Ada Shannon] Closer. DeltaNet is about 19 at 32K, 2.6 points behind, with no test-time training. The average also hides a regression. On the easiest single-needle task at 32K, DeltaNet answers essentially everything and IP-TTCD about one in a hundred. At 4K it gets 48 against In-Place TTT's 96. The wins are on harder, distractor-heavy tasks, and the paper doesn't explain the easy-task collapse. Everything is a single run with no intervals, so 2 to 3 point gaps shouldn't be called reliable.

38
00:09:09,966 --> 00:09:12,288
[Hal Turing] What about perplexity and the synthetic suite?

39
00:09:12,288 --> 00:09:40,012
[Dr. Ada Shannon] Sliding-window perplexity measures a fixed final block as you supply more preceding context. Lower is better. IP-TTCD is lowest at both scales and the advantage grows with context. But the plots span roughly 14 to 16.5 at 340M and 11.5 to 13 at 760M, and the text gives no exact deltas. On MAD it scores 82.1 overall against 78.4, with the biggest gains on fuzzy recall and compression.

40
00:09:40,012 --> 00:09:43,124
[Hal Turing] Now the retrofit. Does it work on models people already have?

41
00:09:43,124 --> 00:10:12,427
[Dr. Ada Shannon] Modestly, and mostly at the far end. On the 1.7B model at 64K, RULER goes from about 14 correct per hundred with In-Place TTT, or 13 with plain continual training, to about 19. On the 8B model the whole-benchmark average is 0.36 points above In-Place TTT and 0.9 above continual training, on a 100-point scale. And at 360M with 64K, IP-TTCD is the lowest of the three.

42
00:10:12,427 --> 00:10:15,492
[Hal Turing] So the headline gain is small where the base model is strongest.

43
00:10:15,492 --> 00:10:25,709
[Dr. Ada Shannon] Yes, and no variance is reported. Under sliding-window inference the three continual-training variants land within about a point of each other. So the benefit depends on running full attention at inference.

44
00:10:25,709 --> 00:10:28,496
[Hal Turing] Do the ablations isolate the distillation idea?

45
00:10:28,496 --> 00:10:57,335
[Dr. Ada Shannon] They isolate the architecture. At 340M, projection plus convolution is best. Convolution alone gets most of the downward trend but at higher perplexity. Projection alone does little, and the authors say they didn't investigate why. More TTT layers helps. Chunk sizes of 512, 1024 and 2048 don't hurt, and 2048 is best. What's missing is any ablation that keeps this architecture and swaps out the teacher-residual target.

46
00:10:57,335 --> 00:11:11,917
[Hal Turing] That brings us back to the first open question. In the from-scratch runs the teacher sees 2048 tokens and the student sees 1024. Is that distilling remote history, or just a modest window extension accumulated over time?

47
00:11:11,917 --> 00:11:33,233
[Dr. Ada Shannon] The second one is what the paper can show. The residual only measures what an extra 1024 tokens of attention would have added. Anything older than 2048 never enters either forward pass. It reaches the model only through state already written. And Proposition 1 assumes the supporting context sits inside the teacher window, so the theory says nothing about content beyond it.

48
00:11:33,233 --> 00:11:35,741
[Hal Turing] Is there a counterpoint in the retrofit experiments?

49
00:11:35,741 --> 00:12:10,803
[Dr. Ada Shannon] Yes. There the teacher is full attention up to 64K against a 2048 student, so the gap is far larger, and the gains do concentrate at the longest lengths. That's suggestive. But that setting also changes the base model, the optimizer and the baselines, so it isn't a controlled test of the ratio. The experiment that would settle it is the same model and budget at 1024/2048, 1024/8192 and 1024/32K. Or a probe that puts the needle beyond the teacher window. Neither is in the paper.

50
00:12:10,803 --> 00:12:18,790
[Hal Turing] Second question. Equation 4 is a distillation loss, but what executes is Equation 12. So is the loss training anything?

51
00:12:18,790 --> 00:12:42,475
[Dr. Ada Shannon] Not directly. The gradient of the loss motivates the shape of the write: residual as value, activation as key. What runs passes the residual through a learned projection and convolutions and scales it with a learned gate. Those parameters are trained by ordinary next-token loss. So whether the write is useful is decided by the outer objective, not by the distillation term.

52
00:12:42,475 --> 00:12:44,982
[Hal Turing] Which means the existing ablations don't answer it.

53
00:12:44,982 --> 00:13:14,843
[Dr. Ada Shannon] Right. They vary the architecture: projection, convolution, layer count, chunk size. The missing control keeps the whole architecture, the two windows, convolutions, projection, scan and four layers, and swaps only the target. Use IP-TTT's LM-aligned target, or a student-only, shuffled or zero residual. If the gain survives, the architecture is doing the work. If it vanishes, the distillation idea earns its name. That control is absent.

54
00:13:14,843 --> 00:13:16,887
[Hal Turing] And the main comparison has other confounds.

55
00:13:16,887 --> 00:13:44,100
[Dr. Ada Shannon] Several. TTCD adds a teacher pass, plus convolutions and a projection, relative to IP-TTT. The IP-TTT baseline is the input-embedding variant because the hidden-state one was unstable, so it isn't a perfectly matched architecture. The retrieval average at 32K is only a couple of points above DeltaNet. The single-needle regression is unexplained. And everything is single-seed with no intervals.

56
00:13:44,100 --> 00:13:46,515
[Hal Turing] What comparisons would change your reading most?

57
00:13:46,515 --> 00:14:22,738
[Dr. Ada Shannon] TTT-E2E, from Tandon and colleagues in 2025, which uses a next-token objective with meta-learned initialization. It's the closest long-context rival, and the paper explicitly leaves it out. Next, the original TTT-Linear and TTT-MLP reconstruction objective from Yu Sun and colleagues in 2024, under identical parameterization. That's the thing TTCD claims to improve on. LaCT appears only on the synthetic suite. Cartridges and KV-Distill are offline compression, so they aren't like-for-like.

58
00:14:22,738 --> 00:14:26,732
[Hal Turing] Now the cost side. Is this practical to bolt onto an existing model?

59
00:14:26,732 --> 00:14:56,082
[Dr. Ada Shannon] The cost is modest where it was measured. With fused kernels, prefill is about 1.3 to 6.6 percent slower than IP-TTT on the newer attention stack. Fusion gives a 10 to 14 percent speedup over the naive two-kernel version. Training is 1.1 to 1.2 times slower than standard baselines. The short-context benchmarks show no sign of harm, though the differences there are a point or less and unreplicated, so I'd only say no sign of harm.

60
00:14:56,082 --> 00:14:57,336
[Hal Turing] What was not measured?

61
00:14:57,336 --> 00:15:07,971
[Dr. Ada Shannon] Decode. Throughput is prefill only on H100, with nothing on decode latency or the memory of the fast-weight state. And from-scratch evidence stops at 760M.

62
00:15:07,971 --> 00:15:09,643
[Hal Turing] So what does the work establish?

63
00:15:09,643 --> 00:15:50,742
[Dr. Ada Shannon] That writing a teacher-minus-student residual into MLP fast weights, through a learned write path, beats IP-TTT and the recurrent baselines on harder long-context retrieval at 760M. It also adds modest long-length gains when retrofitted onto existing models. It doesn't establish that the benefit comes from the distillation target rather than the added architecture. It doesn't establish that it's distilling history beyond a small window gap. And it doesn't show scaling from scratch past 760M. The phrase learning what to remember is an interpretation, because the signal is a local counterfactual within a window gap, not later usefulness.

64
00:15:50,742 --> 00:15:51,857
[Hal Turing] Who should care right now?

65
00:15:51,857 --> 00:16:03,420
[Dr. Ada Shannon] People retrofitting long context onto models they already have, since the layer count and overhead are small and the code is released. But treat the 8B margin as unproven until someone repeats it.

66
00:16:03,420 --> 00:16:05,603
[Hal Turing] And the next experiments are fairly clear.

67
00:16:05,603 --> 00:16:20,185
[Dr. Ada Shannon] The window-ratio sweep and the target-swap control, first. Then seeds with intervals, a TTT-E2E comparison, an explanation of the single-needle failure, scaling past 760M, and decode and memory costs.

68
00:16:20,185 --> 00:16:35,742
[Hal Turing] So the takeaway: a teacher-student gap written into fast weights helps on harder long-context retrieval at small scale, at a small prefill cost. The evidence can't yet say whether the distillation or the extra architecture is responsible. Thanks for listening, and we'll see you next time.