1
00:00:01,000 --> 00:00:07,919
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,919 --> 00:00:47,904
[Hal Turing] Today we're looking at "Training Large Language Models to Reason in a Continuous Latent Space," the Coconut paper. First author is Shibo Hao, with Yuandong Tian and colleagues, from FAIR at Meta and UC San Diego. It first appeared on arXiv in December 2024, and the version we're reading is the fourth, dated August 2026. The question is whether a model that feeds its own last hidden state back in as the next input, instead of writing reasoning as words, reasons better or more cheaply. And whether it picks up something like breadth-first search along the way.

3
00:00:47,904 --> 00:00:57,888
[Dr. Ada Shannon] The claim worth testing is that it learns search without being told to. We'll get to whether the evidence supports that. First, why would anyone want to reason without words?

4
00:00:57,888 --> 00:01:00,675
[Hal Turing] Words are how we reason, and chain-of-thought works.

5
00:01:00,675 --> 00:01:29,746
[Dr. Ada Shannon] It does. Chain-of-thought, from Wei and colleagues at Google in 2022, has the model write intermediate steps before answering. Each generated token loops back in as input, so the model gets more sequential computation than one forward pass allows. Feng and colleagues in 2023, and Li and colleagues in 2024, gave theory for that added depth. The cost is that every step has to collapse into one discrete word. The model picks a token, the rest of the hidden state is discarded, and it has committed.

6
00:01:29,746 --> 00:01:32,300
[Hal Turing] So if the token was wrong, the model is stuck with it.

7
00:01:32,300 --> 00:01:51,991
[Dr. Ada Shannon] Unless it backtracks in text. The authors also argue that most tokens in a reasoning trace exist for fluency, while a few carry the hard planning. Yet the model spends the same compute on each. They add a neuroscience motivation, that human reasoning doesn't seem to run through the language network. I'd treat that as motivation, not evidence for the method.

8
00:01:51,991 --> 00:01:55,288
[Hal Turing] Fair. But extra compute without words already existed, didn't it?

9
00:01:55,288 --> 00:02:31,047
[Dr. Ada Shannon] Yes, and that matters for what Coconut can claim. Pause tokens, from Goyal and colleagues at Google in 2023, are learnable filler tokens inserted between question and answer. They give the model extra forward passes, but they carry no recycled state. Pfau and colleagues in 2024 showed that even plain filler tokens, like repeated dots, can help on some tasks. Then iCoT, by Deng, Choi and Shieber in 2024, trains on written reasoning and gradually removes the reasoning tokens until the model answers directly. No latent vectors at all.

10
00:02:31,047 --> 00:02:36,527
[Hal Turing] So the real question is what feeding the hidden state back adds on top of extra compute and gradual removal.

11
00:02:36,527 --> 00:03:01,047
[Dr. Ada Shannon] Exactly. Here's the mechanism. In Coconut, the final-layer vector at a position, the last hidden state, is not decoded into a word. It goes straight back in as the next input embedding. The paper calls it a continuous thought. Special tokens, bot and eot, mark where the latent stretch begins and ends, and after that the model writes words again.

12
00:03:01,047 --> 00:03:05,366
[Hal Turing] How does it learn to do that? Nobody labels what a good thought looks like.

13
00:03:05,366 --> 00:03:27,193
[Dr. Ada Shannon] Right, there's no supervision on the thoughts. Training uses a curriculum, meaning stages of increasing difficulty, borrowed from iCoT. It starts from ordinary written reasoning, then progressively swaps written steps for latent ones. The details come in a moment. The key point is that the loss only sees the words after the thoughts.

14
00:03:27,193 --> 00:03:29,840
[Hal Turing] Okay, now the search part. I want a concrete example.

15
00:03:29,840 --> 00:03:54,546
[Dr. Ada Shannon] The paper builds a dataset called ProsQA. Each question has a random graph of made-up concepts. Every terpus is a brimpus, every brimpus is a lempus, and so on. Then it asks, is Tom a lempus or a scrompus? Exactly one is reachable from Tom. The graphs include distracting branches. Breadth-first search means expanding every neighbor at one level before going deeper, so you keep all the branches open.

16
00:03:54,546 --> 00:03:57,657
[Hal Turing] Whereas a model writing words picks one edge and walks down it.

17
00:03:57,657 --> 00:04:24,778
[Dr. Ada Shannon] And if that branch dead-ends, it has to notice and back up, or it fabricates an edge to reach the answer. The authors argue a continuous thought can hold several candidates at once and prune them over later steps. I'll call that the latent tree search view. Their evidence is that you can force a thought through the word head and read off probabilities over concepts, an implicit value for each candidate. That's an interpretation of a trained model, not something they train for.

18
00:04:24,778 --> 00:04:26,729
[Hal Turing] How does that differ from Tree of Thoughts?

19
00:04:26,729 --> 00:04:45,119
[Dr. Ada Shannon] Tree of Thoughts, Yao and colleagues in 2023, runs the search outside the model. It samples candidate thoughts, prompts a model to score them, and prunes. Every step costs tokens. Coconut would do it inside one forward chain. Later theory work formalizes that, and we'll get to it.

20
00:04:45,119 --> 00:04:46,791
[Hal Turing] Two other benchmarks show up too.

21
00:04:46,791 --> 00:05:06,621
[Dr. Ada Shannon] ProntoQA is an earlier synthetic logic benchmark with less branching than ProsQA. GSM8k is grade-school math word problems, and it's the only natural-language task here. All three are scored by final-answer accuracy, higher is better. Keep in mind that ProsQA is the authors' own dataset, built to favor search.

22
00:05:06,621 --> 00:05:11,450
[Hal Turing] Back to the mechanism. Why does skipping the word head matter for training?

23
00:05:11,450 --> 00:05:34,299
[Dr. Ada Shannon] Picking a token is a discrete step, so gradients can't pass through it. A vector is differentiable, so the loss can flow back through the whole chain of thoughts with ordinary backpropagation. The cost is sequential compute. With n thoughts, training needs n+1 forward passes, each waiting on the one before. The authors flag that as an open efficiency problem.

24
00:05:34,299 --> 00:05:38,246
[Hal Turing] But a model can't just discover what to put in those vectors from answers alone.

25
00:05:38,246 --> 00:06:03,556
[Dr. Ada Shannon] Right, and the paper agrees. It starts from ordinary language chain-of-thought data. At stage k, the first k language steps are replaced by k times c thoughts, where c is thoughts per step. Loss is masked on the question and on the thoughts, and the optimizer resets at each stage change. Nothing asks a thought to summarize the sentence it replaced. It only has to help predict what comes next.

26
00:06:03,556 --> 00:06:04,903
[Hal Turing] What are the actual settings?

27
00:06:04,903 --> 00:06:22,271
[Dr. Ada Shannon] ProsQA and ProntoQA use one thought per step and six stages, five epochs each, then stay in the final stage up to fifty epochs, keeping the best validation checkpoint. GSM8k uses two thoughts per step and three stages, plus a final stage with all language removed.

28
00:06:22,271 --> 00:06:24,825
[Hal Turing] At test time, does the model decide how long to think?

29
00:06:24,825 --> 00:06:43,448
[Dr. Ada Shannon] No, the number of thoughts is fixed and padded in, with greedy decoding. They tried a learned stopping classifier and say it worked comparably. The useful trick is that the same weights can be forced to use zero through six thoughts, then finish in language. That gives a dial from fully language reasoning to fully latent.

30
00:06:43,448 --> 00:06:45,723
[Hal Turing] Then the headline. How does it do on ProsQA?

31
00:06:45,723 --> 00:07:07,736
[Dr. Ada Shannon] Out of a hundred questions, language chain-of-thought gets about 77 right and Coconut about 97. That's GPT-2 on the authors' own synthetic dataset. Coconut also uses about 14 tokens against 49, and at batch size one on an A100 it takes 0.15 seconds per question against 0.47. Roughly a third of the time.

32
00:07:07,736 --> 00:07:09,826
[Hal Turing] What does skipping reasoning entirely score?

33
00:07:09,826 --> 00:07:32,442
[Dr. Ada Shannon] About 77, the same as chain-of-thought. On a task built to need search, written-out reasoning bought almost nothing. Hold that thought, because there's a puzzle coming. The paper also runs ablation variants. One removes the curriculum. One keeps the curriculum but inserts no thoughts. One keeps it and inserts pause tokens where thoughts would go. We'll come back to what those show.

34
00:07:32,442 --> 00:07:34,578
[Hal Turing] Does the dial show anything about the errors?

35
00:07:34,578 --> 00:07:56,405
[Dr. Ada Shannon] Qualitatively, yes. Chain-of-thought often hallucinates an edge that doesn't exist, or ends at the wrong target. Their case study shows it hitting a dead end and inventing a link. As you force more latent thoughts, hallucinations and wrong-target endings fall, and correct paths rise. The paper doesn't give the exact percentages in the text, so I'll stay qualitative.

36
00:07:56,405 --> 00:07:58,355
[Hal Turing] So where does the search claim come from?

37
00:07:58,355 --> 00:08:19,903
[Dr. Ada Shannon] The decoding probe. In their worked example, 'lempus' leads at the first step with a value of 0.33. At the second step the top value, 0.87, goes to 'rorpus', a child of 'grimpus'. A greedy model would have followed lempus. Instead it kept other branches alive.

38
00:08:19,903 --> 00:08:21,900
[Hal Turing] Is that one example, or a pattern?

39
00:08:21,900 --> 00:08:41,684
[Dr. Ada Shannon] One worked example, plus aggregates. The top three candidates' values are spread out at the first thought and the gap narrows at the second, so exploration seems to narrow. And nodes near the leaves get sharper, more accurate values than nodes far away. That's their explanation for why delaying commitment helps. It's a coherent story, but it's correlational.

40
00:08:41,684 --> 00:08:44,609
[Hal Turing] What about natural language? Math isn't a graph.

41
00:08:44,609 --> 00:09:08,898
[Dr. Ada Shannon] That's the less flattering result. Chain-of-thought gets about 43 out of a hundred on GSM8k. Coconut gets about 34. So latent reasoning is cheaper, with about 8 tokens against 25, but weaker. Still, the same curriculum with pause tokens gets 24, and with nothing inserted it gets under 22. That's a 10 to 12 point gap, so here the vectors seem to matter.

42
00:09:08,898 --> 00:09:10,848
[Hal Turing] Anything inside the thoughts that you can read?

43
00:09:10,848 --> 00:09:32,025
[Dr. Ada Shannon] The first thought often decodes to a plausible intermediate variable in the calculation. And on the efficiency curve, when language-only models internalize more steps, accuracy drops fast. Coconut degrades much more gently. The caveat is that the iCoT number on GSM8k, 30, comes from the iCoT paper, not a rerun.

44
00:09:32,025 --> 00:09:40,291
[Hal Turing] So, the puzzle. Pause-as-thought, without-thought, iCoT. How much of the ProsQA headline survives them?

45
00:09:40,291 --> 00:10:01,142
[Dr. Ada Shannon] Not the part about latent vectors. The variants with no continuous thoughts land at 95.5, 96.6 and 98.2. Coconut is at 97.0. That's a spread of under three points, and one of the no-latent models is higher. So the 20-point gain over language CoT can't be credited to feeding hidden states back.

46
00:10:01,142 --> 00:10:02,350
[Hal Turing] And the curriculum row?

47
00:10:02,350 --> 00:10:22,226
[Dr. Ada Shannon] Take the curriculum away and Coconut falls to 76.1, which is the no-reasoning score. So the curriculum is necessary. The without-thought row says it's nearly sufficient. The paper concedes some of this. It says computational capacity may not be the bottleneck on these synthetic tasks, and that iCoT's finer schedule probably helps.

48
00:10:22,226 --> 00:10:24,920
[Hal Turing] Then what's left of the breadth-first-search story?

49
00:10:24,920 --> 00:10:54,177
[Dr. Ada Shannon] Two separate claims. One: inside a trained Coconut model, more latent thoughts give better answers, and the decoded values look like parallel exploration. The other: you need latent vectors to get this accuracy. The paper supports the first, with limits. It doesn't support the second on ProsQA, because iCoT gets there with 8.2 tokens and no latent vectors. And nobody probed how iCoT or the pause variants solve it internally.

50
00:10:54,177 --> 00:10:56,220
[Hal Turing] How solid is the first claim, though?

51
00:10:56,220 --> 00:11:36,251
[Dr. Ada Shannon] Thinner than it reads. The worked example is one graph. The value is the probability of a concept word after forcing the model back into language mode. That reads the vector through the word head, so it shows what a language readout contains, not that the vector holds a frontier of nodes. The height-versus-value plot is correlational too, and in graphs of about 23 nodes, nodes near leaves may be easy to score for trivial reasons. A degree-and-depth baseline would tell us. Better still, run the same probe on the without-thought and pause models, and try an intervention: remove the branch information from the vector and see whether accuracy drops.

52
00:11:36,251 --> 00:11:43,682
[Hal Turing] The CoT baseline bothers me. 77.5 against 76.7 for no reasoning, on a task built to need search.

53
00:11:43,682 --> 00:12:25,989
[Dr. Ada Shannon] It should bother you. The text doesn't say whether CoT got the same fifty epochs, the same checkpoint selection, or any tuning. Figure 4 shows it hallucinating an edge, which may be a weakly supervised 124-million-parameter model rather than a limit of language. The fairer comparison is language search done properly: Stream of Search, by Kanishk Gandhi at Stanford in 2024, trains on search traces with backtracking written as text. Beyond A*, by Lucas Lehnert and colleagues at Meta in 2024, trains on A-star traces. Neither was run on ProsQA. Also, ProsQA has 500 test questions, and the number of seeds behind the plus-or-minus values isn't stated.

54
00:12:25,989 --> 00:12:29,564
[Hal Turing] So is there any place where the latent loop clearly earns its keep?

55
00:12:29,564 --> 00:12:42,010
[Dr. Ada Shannon] GSM8k. Coconut beats the pause and no-thought controls there by about ten points, so the loop adds something on that task. But it still loses to CoT, and the iCoT number was copied from Deng's paper, not rerun.

56
00:12:42,010 --> 00:12:42,846
[Hal Turing] And scale?

57
00:12:42,846 --> 00:13:21,995
[Dr. Ada Shannon] The gain shrinks. On GSM8k, Coconut beats no-reasoning by 5.7 points on Llama 3.2-3B and 1.4 on Llama 3-8B. There's no CoT baseline for either model and no variance reported, so don't extrapolate in either direction. The authors blame heavy language pretraining and call for latent-space pretraining, which is a hypothesis. There's also the training cost. Every thought needs its own sequential forward pass, and the paper never benchmarks that against CoT training. They also report training instability at three thoughts per step on GSM8k, which they treat as a training problem.

58
00:13:21,995 --> 00:13:24,178
[Hal Turing] Is there theory that backs the search idea?

59
00:13:24,178 --> 00:13:59,379
[Dr. Ada Shannon] Zhu, Hao, Hu, Jiao, Russell and Tian, 2025, "Reasoning by Superposition." They construct a small transformer whose continuous thoughts hold a superposition of reachable nodes. It solves directed-graph reachability in steps proportional to the graph's diameter, which discrete CoT can't match. That's a stylized construction, not the trained GPT-2. Their follow-up, "Emergence of Superposition," studies how it arises under the Coconut objective. That bears on whether search emerges or is induced by the curriculum, though I won't claim more than that.

60
00:13:59,379 --> 00:14:03,187
[Hal Turing] What about oversight? If the reasoning isn't words, nobody can read it.

61
00:14:03,187 --> 00:14:32,119
[Dr. Ada Shannon] The paper doesn't discuss that. Turpin and colleagues showed in 2024 that written chains of thought can already be unfaithful, but they're at least readable. Dense vectors remove even that. The decode-to-word probe helps a little, but nobody has shown it's faithful to what the model uses. It's also unclear how you'd do outcome-reward reinforcement learning on latent thoughts without sampling. And the curriculum needs gold language chains, so this isn't reasoning without language supervision.

62
00:14:32,119 --> 00:14:33,373
[Hal Turing] Where does it go from here?

63
00:14:33,373 --> 00:15:04,813
[Dr. Ada Shannon] The authors name combining Coconut with iCoT's token-level schedule, latent pretraining, a language skeleton with latent completion, and finer stage schedules. Related work points the same way: Geiping's recurrent-depth model, Barrault's Large Concept Models, and Gladstone's Energy-Based Transformers. Practically, the credible use today is a small model on a search-heavy task, with roughly a third of the wall-clock per question at batch size 1 on an A100. Whether that holds at large batch sizes is untested.

64
00:15:04,813 --> 00:15:25,386
[Hal Turing] So, summing up. A curriculum that swaps language reasoning for recycled hidden states can train a small model to match or beat CoT on synthetic graph search, using far fewer tokens. And a trained Coconut model's thoughts carry usable information about several candidate paths.

65
00:15:25,386 --> 00:15:41,826
[Dr. Ada Shannon] Right. What it doesn't show is that the latent vectors produce the ProsQA gain, that breadth-first search is required, or that this beats CoT on natural-language math or on larger models. It's a proof of concept with an interpretation that still needs its causal test.

66
00:15:41,826 --> 00:15:45,169
[Hal Turing] That's our time. Thanks for listening, and we'll see you next time.