Coconut: Reasoning in Continuous Latent Space Instead of Words

This episode examines the Coconut paper, which trains a language model to feed its last hidden state back in as the next input, so it reasons in continuous "thoughts" instead of written words. It first covers why chain-of-thought forces every step through a single discrete token, then sets Coconut against nearby approaches. Those are pause tokens, filler tokens, iCoT's gradual removal of reasoning tokens, and Tree of Thoughts, which runs its search outside the model. The discussion explains the curriculum that swaps written steps for latent ones, and why a vector rather than a token lets gradients flow through the whole chain. It also notes the cost of n+1 sequential forward passes in training. The central claim is that latent thoughts can hold several candidates at once, like breadth-first search, tested on the authors' own ProsQA graph-reasoning dataset alongside ProntoQA and GSM8k. The hosts stay skeptical, pointing out that ProsQA was built to favor search and that the search interpretation is read off a trained model rather than trained for.

Interactive Visualization: Coconut: Reasoning in Continuous Latent Space Instead of Words

Sources:
1. Training Large Language Models to Reason in a Continuous Latent Space — Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian, 2024
http://arxiv.org/abs/2412.06769
2. Training Large Language Models to Reason in a Continuous Latent Space — Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian, 2024 (arXiv 2412.06769; revised through 2026)
https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space
3. Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought — Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, Yuandong Tian, 2025
https://scholar.google.com/scholar?q=Reasoning+by+Superposition%3A+A+Theoretical+Perspective+on+Chain+of+Continuous+Thought
4. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan, 2023
https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models
5. Reasoning with Language Model is Planning with World Model (RAP) — Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu, 2023
https://scholar.google.com/scholar?q=Reasoning+with+Language+Model+is+Planning+with+World+Model+%28RAP%29
6. From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step — Yuntian Deng, Yejin Choi, Stuart Shieber, 2024
https://scholar.google.com/scholar?q=From+Explicit+CoT+to+Implicit+CoT%3A+Learning+to+Internalize+CoT+Step+by+Step
7. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein, 2025
https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach
8. Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought — Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, Yuandong Tian, 2025
https://scholar.google.com/scholar?q=Emergence+of+Superposition%3A+Unveiling+the+Training+Dynamics+of+Chain+of+Continuous+Thought
9. Think before you speak: Training language models with pause tokens — Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan, 2023
https://scholar.google.com/scholar?q=Think+before+you+speak%3A+Training+language+models+with+pause+tokens
10. Let's Think Dot by Dot: Hidden Computation in Transformer Language Models — Jacob Pfau, William Merrill, Samuel R. Bowman, 2024
https://scholar.google.com/scholar?q=Let%27s+Think+Dot+by+Dot%3A+Hidden+Computation+in+Transformer+Language+Models
11. Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping — Lucas Lehnert, Sainbayar Sukhbaatar, Paul McVay, Michael Rabbat, Yuandong Tian, 2024
https://scholar.google.com/scholar?q=Beyond+A%2A%3A+Better+Planning+with+Transformers+via+Search+Dynamics+Bootstrapping
12. Stream of Search (SoS): Learning to Search in Language — Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, Noah D. Goodman, 2024
https://scholar.google.com/scholar?q=Stream+of+Search+%28SoS%29%3A+Learning+to+Search+in+Language
13. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Miles Turpin, Julian Michael, Ethan Perez, Samuel Bowman, 2024
https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think%3A+Unfaithful+Explanations+in+Chain-of-Thought+Prompting
14. Large Concept Models: Language Modeling in a Sentence Representation Space — Loïc Barrault et al., 2024
https://scholar.google.com/scholar?q=Large+Concept+Models%3A+Language+Modeling+in+a+Sentence+Representation+Space
15. Energy-Based Transformers are Scalable Learners and Thinkers — Alexi Gladstone et al., 2025
https://scholar.google.com/scholar?q=Energy-Based+Transformers+are+Scalable+Learners+and+Thinkers
16. Recurrent Memory Transformer — Aydar Bulatov, Yury Kuratov, Mikhail Burtsev, 2022
https://scholar.google.com/scholar?q=Recurrent+Memory+Transformer
Interactive Visualization: Coconut: Reasoning in Continuous Latent Space Instead of Words