Stealing Hidden Reasoning Traces from Proprietary LLM APIs
Oct 2, 2026
This episode explores a paper showing that the encrypted reasoning blocks returned by proprietary LLM APIs can be decoded by replaying them into a cheaper, less safeguarded sibling model from the same provider. It explains why providers hide chain of thought, and how stateless APIs make clients hold the opaque blob. The discussion then covers the flaw: an integrity tag proves a block is intact, but not that it belongs to the caller, so blocks appear to be valid across sessions, users and models under what looks like a single global key. The episode walks through the decryption-jailbreak method against Claude, GPT and Gemini models, including prefilling, many sampled prompt templates, chunking output to dodge a roughly fifty-token copy filter, and the extraction-ratio check for faithfulness. It also previews the four attacks: circumventing anti-distillation, leaking private data from shared logs, pulling hazardous content that the visible answer refuses, and invisible prompt injection that plaintext monitors cannot see.
Sources:
1. Stealing Reasoning Traces from Proprietary LLM APIs — Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping, Maksym Andriushchenko, 2026
http://arxiv.org/abs/2608.09867
2. Stealing Part of a Production Language Model — Nicholas Carlini, Daniel Paleka, Krishnamurthy Dvijotham, et al., 2024
https://scholar.google.com/scholar?q=Stealing+Part+of+a+Production+Language+Model
3. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — Cheng-Yu Hsieh et al., 2023
https://scholar.google.com/scholar?q=Distilling+Step-by-Step%21+Outperforming+Larger+Language+Models+with+Less+Training+Data+and+Smaller+Model+Sizes
4. s1: Simple test-time scaling — Niklas Muennighoff et al., 2025
https://scholar.google.com/scholar?q=s1%3A+Simple+test-time+scaling
5. Reasoning Models Don't Always Say What They Think — Yanda Chen et al. (Anthropic), 2025
https://scholar.google.com/scholar?q=Reasoning+Models+Don%27t+Always+Say+What+They+Think
6. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — Tomek Korbak et al., 2025
https://scholar.google.com/scholar?q=Chain+of+Thought+Monitorability%3A+A+New+and+Fragile+Opportunity+for+AI+Safety
7. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation — Bowen Baker et al., 2025
https://scholar.google.com/scholar?q=Monitoring+Reasoning+Models+for+Misbehavior+and+the+Risks+of+Promoting+Obfuscation
8. Extracting Training Data from Large Language Models — Nicholas Carlini et al., 2021
https://scholar.google.com/scholar?q=Extracting+Training+Data+from+Large+Language+Models
9. Ignore Previous Prompt: Attack Techniques For Language Models — Fábio Perez, Ian Ribeiro, 2022
https://scholar.google.com/scholar?q=Ignore+Previous+Prompt%3A+Attack+Techniques+For+Language+Models
10. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Kai Greshake et al., 2023
https://scholar.google.com/scholar?q=Not+what+you%27ve+signed+up+for%3A+Compromising+Real-World+LLM-Integrated+Applications+with+Indirect+Prompt+Injection
11. Alignment faking / Stress Testing Deliberative Alignment for Anti-Scheming Training — Schoen et al. (OpenAI / Apollo), 2025
https://scholar.google.com/scholar?q=Alignment+faking+%2F+Stress+Testing+Deliberative+Alignment+for+Anti-Scheming+Training
1. Stealing Reasoning Traces from Proprietary LLM APIs — Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping, Maksym Andriushchenko, 2026
http://arxiv.org/abs/2608.09867
2. Stealing Part of a Production Language Model — Nicholas Carlini, Daniel Paleka, Krishnamurthy Dvijotham, et al., 2024
https://scholar.google.com/scholar?q=Stealing+Part+of+a+Production+Language+Model
3. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — Cheng-Yu Hsieh et al., 2023
https://scholar.google.com/scholar?q=Distilling+Step-by-Step%21+Outperforming+Larger+Language+Models+with+Less+Training+Data+and+Smaller+Model+Sizes
4. s1: Simple test-time scaling — Niklas Muennighoff et al., 2025
https://scholar.google.com/scholar?q=s1%3A+Simple+test-time+scaling
5. Reasoning Models Don't Always Say What They Think — Yanda Chen et al. (Anthropic), 2025
https://scholar.google.com/scholar?q=Reasoning+Models+Don%27t+Always+Say+What+They+Think
6. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — Tomek Korbak et al., 2025
https://scholar.google.com/scholar?q=Chain+of+Thought+Monitorability%3A+A+New+and+Fragile+Opportunity+for+AI+Safety
7. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation — Bowen Baker et al., 2025
https://scholar.google.com/scholar?q=Monitoring+Reasoning+Models+for+Misbehavior+and+the+Risks+of+Promoting+Obfuscation
8. Extracting Training Data from Large Language Models — Nicholas Carlini et al., 2021
https://scholar.google.com/scholar?q=Extracting+Training+Data+from+Large+Language+Models
9. Ignore Previous Prompt: Attack Techniques For Language Models — Fábio Perez, Ian Ribeiro, 2022
https://scholar.google.com/scholar?q=Ignore+Previous+Prompt%3A+Attack+Techniques+For+Language+Models
10. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Kai Greshake et al., 2023
https://scholar.google.com/scholar?q=Not+what+you%27ve+signed+up+for%3A+Compromising+Real-World+LLM-Integrated+Applications+with+Indirect+Prompt+Injection
11. Alignment faking / Stress Testing Deliberative Alignment for Anti-Scheming Training — Schoen et al. (OpenAI / Apollo), 2025
https://scholar.google.com/scholar?q=Alignment+faking+%2F+Stress+Testing+Deliberative+Alignment+for+Anti-Scheming+Training