Invisible Reasoning: Filler Tokens Boost LLM Accuracy Unseen

This episode examines a paper arguing that frontier LLMs can do useful reasoning that never appears in their chain-of-thought. Inserting an identical filler string, such as "count one to a hundred," between the question and the answer raises accuracy on tasks like 4-digit multiplication, nested arithmetic, and variable counting across 13 models. The discussion separates this "invisible reasoning" from unfaithful explanations and steganography, since here the tokens carry nothing to decode. It contrasts the finding with earlier dummy-token and pause-token work, which needed special training, and notes the authors' hedge that the uplift may not be extra computation alone. It also covers the safety stakes for chain-of-thought monitoring, the efficiency incentives for labs, and which filler types help or hurt across models. Listeners interested in AI oversight and what reading a model's reasoning can actually reveal will find it useful.

Sources:
1. Not All LLM Reasoning is Visible in the Chain-of-Thought — Vatsal Baherwani, Tom Goldstein, Ashwinee Panda, 2026
http://arxiv.org/abs/2607.22925
2. Let's Think Dot by Dot: Hidden Computation in Transformer Language Models — Jacob Pfau, William Merrill, Samuel R. Bowman, 2024
https://scholar.google.com/scholar?q=Let%27s+Think+Dot+by+Dot%3A+Hidden+Computation+in+Transformer+Language+Models
3. Think before you speak: Training Language Models With Pause Tokens — Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan, 2023 (ICLR 2024)
https://scholar.google.com/scholar?q=Think+before+you+speak%3A+Training+Language+Models+With+Pause+Tokens
4. The Expressive Power of Transformers with Chain of Thought — William Merrill, Ashish Sabharwal, 2023 (ICLR 2024)
https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought
5. Training Large Language Models to Reason in a Continuous Latent Space (Coconut) — Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, Yuandong Tian, 2024
https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space+%28Coconut%29
6. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, and many co-authors from multiple labs and institutes, 2025
https://scholar.google.com/scholar?q=Chain+of+Thought+Monitorability%3A+A+New+and+Fragile+Opportunity+for+AI+Safety
7. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation — Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, David Farhi, 2025
https://scholar.google.com/scholar?q=Monitoring+Reasoning+Models+for+Misbehavior+and+the+Risks+of+Promoting+Obfuscation
8. Reasoning Models Don't Always Say What They Think — Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, Ethan Perez, 2025
https://scholar.google.com/scholar?q=Reasoning+Models+Don%27t+Always+Say+What+They+Think
9. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman, 2023
https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think%3A+Unfaithful+Explanations+in+Chain-of-Thought+Prompting
10. Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance — Ryan Greenblatt, 2025
https://scholar.google.com/scholar?q=Recent+LLMs+can+use+filler+tokens+or+problem+repeats+to+improve+%28no-CoT%29+math+performance
11. Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoning — Zeru Shi, Yingjia Wan, Zhenting Wang, Qifan Wang, Fan Yang, Elisa Kreiss, Ruixiang Tang, 2025
https://scholar.google.com/scholar?q=Meaningless+Tokens%2C+Meaningful+Gains%3A+How+Activation+Shifts+Enhance+LLM+Reasoning
12. Chain of Thought Empowers Transformers to Solve Inherently Serial Problems — Zhiyuan Li, Hong Liu, Denny Zhou, Tengyu Ma, 2024
https://scholar.google.com/scholar?q=Chain+of+Thought+Empowers+Transformers+to+Solve+Inherently+Serial+Problems
13. Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability — Artur Zolkowski, Wen Xing, David Lindner, Florian Tramèr, Erik Jenner, 2025
https://scholar.google.com/scholar?q=Can+Reasoning+Models+Obfuscate+Reasoning%3F+Stress-Testing+Chain-of-Thought+Monitorability
14. Lessons from Studying Two-Hop Latent Reasoning — Mikita Balesni, Tomek Korbak, Owain Evans, 2025
https://scholar.google.com/scholar?q=Lessons+from+Studying+Two-Hop+Latent+Reasoning
15. Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought — Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, Jack Merullo, 2026
https://scholar.google.com/scholar?q=Reasoning+Theater%3A+Disentangling+Model+Beliefs+from+Chain-of-Thought
16. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein, 2025
https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach
17. Auditing Language Models for Hidden Objectives — Samuel Marks et al., 2025
https://scholar.google.com/scholar?q=Auditing+Language+Models+for+Hidden+Objectives
18. Preventing Language Models from Hiding Their Reasoning — Fabien Roger, Ryan Greenblatt, 2023
https://scholar.google.com/scholar?q=Preventing+Language+Models+from+Hiding+Their+Reasoning