1
00:00:01,000 --> 00:00:08,755
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,755 --> 00:00:35,226
[Hal Turing] Today's paper is "Stealing Reasoning Traces from Proprietary LLM APIs," posted to arXiv in August 2026. The first author is Alexander Panfilov, working with colleagues at MATS Research, the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, and collaborators. The claim is that you can read a frontier model's hidden chain of thought by asking its cheaper sibling to decode it.

3
00:00:35,226 --> 00:00:57,331
[Dr. Ada Shannon] And the strong model is never attacked directly. The authors take an encrypted reasoning block from a strong model and hand it to a weaker model from the same provider, which reads it back in plaintext. So the question is this: if any model in a provider's ecosystem can replay these encrypted blocks, can a less safeguarded sibling expose the hidden reasoning, and what does that expose?

4
00:00:57,331 --> 00:01:01,743
[Hal Turing] First, why hide the reasoning at all? The final answer is what the user wants.

5
00:01:01,743 --> 00:01:38,430
[Dr. Ada Shannon] Because the reasoning is denser and more sensitive than the answer. It holds intermediate hypotheses, tool outputs, user data, and context the model never repeats in its reply. Providers hide it for three reasons. One is intellectual property. Another is anti-distillation: if competitors can't see the chain of thought, they can't train student models on it. The third is limiting harmful or sensitive leakage. Anti-distillation includes refusal training against revealing reasoning. On GPT it also includes an API-side filter that rejects outputs copying more than about fifty tokens of the original reasoning.

6
00:01:38,430 --> 00:01:42,239
[Hal Turing] But the model still needs its reasoning on the next turn. Where does it live?

7
00:01:42,239 --> 00:01:57,703
[Dr. Ada Shannon] With you. The API returns an opaque base64 blob, called a signature or encrypted content depending on the provider. The client stores it and sends it back on every turn. The server keeps no state. It decrypts the blob and carries on.

8
00:01:57,703 --> 00:01:59,793
[Hal Turing] What stops me from editing the blob?

9
00:01:59,793 --> 00:02:17,115
[Dr. Ada Shannon] Presumably an AEAD envelope. In plain terms, that's ciphertext plus a tag. If anyone alters the contents or the associated header, the tag check fails and the server rejects it. One caveat: providers haven't documented the scheme. The paper infers it from behavior.

10
00:02:17,115 --> 00:02:21,248
[Hal Turing] So the tag proves the blob is intact. Does it prove the blob is yours?

11
00:02:21,248 --> 00:02:46,744
[Dr. Ada Shannon] That's the flaw. The paper describes reasoning compatibility in three escalating forms. Blocks can be reordered or reused within a session and across sessions. A block from another user's session can be replayed. And a block from one model can be replayed into another. Green had already disclosed portability like this in 2026. The evidence here points to a single global key per provider, so a valid block is valid for anyone.

12
00:02:46,744 --> 00:02:49,530
[Hal Turing] And that matters because the siblings aren't equally defended.

13
00:02:49,530 --> 00:03:14,282
[Dr. Ada Shannon] Right. Frontier models get the heavy anti-distillation training. Cheaper siblings like Haiku 4.5, GPT-5.6 Luna, and Gemini Robotics 1.6 don't get the same treatment. So an unprivileged API user sends a stolen block to the weak model and asks it to transcribe the reasoning attached to it. The authors call this a decryption jailbreak. The jailbroken model is the weak one.

14
00:03:14,282 --> 00:03:18,090
[Hal Turing] So the weak model treats the blob as its own earlier thought and just writes it out.

15
00:03:18,090 --> 00:03:28,075
[Dr. Ada Shannon] That's the intuition. The word "fuzzy" in fuzzy decoder matters, though. The output is sampled, so there's no guarantee of an exact copy. That gap will matter later.

16
00:03:28,075 --> 00:03:29,979
[Hal Turing] Who's the attacker in this threat model?

17
00:03:29,979 --> 00:03:44,097
[Dr. Ada Shannon] Two cases. A first-party attacker has API access and wants reasoning from the provider's own strongest model. A third-party attacker never touches the victim's account. They find blocks in logs the victim published and decode them.

18
00:03:44,097 --> 00:03:45,583
[Hal Turing] So what are the four attacks?

19
00:03:45,583 --> 00:04:06,156
[Dr. Ada Shannon] Here's the roadmap. First, circumventing anti-distillation. Second, extracting private data from shared logs. Third, pulling hazardous content out of reasoning even when the visible answer refuses. Fourth, invisible prompt injection, where the malicious instruction sits inside an encrypted block that no plaintext monitor can read.

20
00:04:06,156 --> 00:04:12,518
[Hal Turing] Start with the first. How do you decode a block you can't read, without querying the model that made it?

21
00:04:12,518 --> 00:04:31,280
[Dr. Ada Shannon] You capture a signed block from Opus 4.8, then replay it into Haiku 4.5 with one fixed prompt: transcribe the reasoning inside thinking-copy tags. You also prefill the assistant turn. Haiku treats the block as its own earlier thought and writes it out. Opus is never asked for the trace.

22
00:04:31,280 --> 00:04:33,509
[Hal Turing] So the cheap model is the only one under pressure.

23
00:04:33,509 --> 00:05:05,274
[Dr. Ada Shannon] Yes. And Claude is the easy case: one prompt, temperature 1, and a keyword filter to drop refusals. GPT-5.6 Luna is harder. They use many prompt templates and sample up to fifty candidates. They chunk the output under roughly fifty tokens, because the API rejects long verbatim copies. And they fall back to the larger Terra model when Luna fails. Gemini Robotics 1.6 is the decoder there because it accepts both 2.5 and 3.x traces.

24
00:05:05,274 --> 00:05:07,224
[Hal Turing] What does the compatibility table show?

25
00:05:07,224 --> 00:05:31,419
[Dr. Ada Shannon] A pattern. Fable 5 blocks can't be replayed at all. GPT-5.6 will replay older GPT blocks. Gemini is fully interchangeable. Placement matters too. Current-turn injection puts the block in the in-progress assistant turn, so the model continues from it. Past-turn injection puts it in an earlier turn. Which one works depends on the model generation.

26
00:05:31,419 --> 00:05:33,834
[Hal Turing] How do they know the decoded text is faithful?

27
00:05:33,834 --> 00:05:58,726
[Dr. Ada Shannon] They compute an extraction ratio: the re-encoded token count of the extracted text divided by the thinking tokens the API billed. Near one is read as faithful. On 120 Codeforces problems, the paper says the two track closely for most inputs. They also evaluate on AIME 2025 and HLE. For Claude there's an optional step where Opus merges up to three noisy Haiku transcriptions into one.

28
00:05:58,726 --> 00:06:00,398
[Hal Turing] And why would anyone want these traces?

29
00:06:00,398 --> 00:06:35,553
[Dr. Ada Shannon] Distillation. The argument is that output-only supervision is thin and trace supervision is richer. They cite Zhang and colleagues, 2026, who trained Qwen2.5-7B on approximate reconstructed traces from GPT-5.4 mini. Accuracy on MATH500 went from 68.4 to 76.0 percent. The paper's cost estimate is about seven hundred twenty dollars to decode ten thousand traces of twelve thousand tokens each, at Haiku 4.5 rates. Roughly a hundred twenty for input and six hundred for output.

30
00:06:35,553 --> 00:06:37,829
[Hal Turing] Is there evidence that someone already did this?

31
00:06:37,829 --> 00:07:10,244
[Dr. Ada Shannon] One weak signal, and the appendix says so itself. Prefill just the first one percent of a decoded Opus trace into the open-weight model Kimi-K3, and its phrasing overlap with Opus's visible answer rises from about sixteen to thirty percent on science problems. Higher means more Opus-like wording. It rose on 29 of 30 problems, and a control model barely moved. But this measures style overlap on thirty problems, run after the patch. It can't establish distillation.

32
00:07:10,244 --> 00:07:13,401
[Hal Turing] Then the safety angle. Does the decoder bypass refusals?

33
00:07:13,401 --> 00:07:29,145
[Dr. Ada Shannon] The reasoning channel does. With a paraphrased HarmBench car-theft prompt, the visible Opus answer is benign, but the decoded reasoning contains the harmful content. In another case the visible answer is just the word Ready, while the reasoning holds the worked chemistry.

34
00:07:29,145 --> 00:07:31,606
[Hal Turing] What about shared agent logs? Those are public.

35
00:07:31,606 --> 00:07:57,659
[Dr. Ada Shannon] They decoded blocks from 6,708 public trajectories. About one session in twenty leaked at least one real sensitive item, including API keys, passwords, tokens and private keys. In 64 cases the secret never appeared in the visible chat. These are counts of distinct artifacts, not confirmed live credentials, and much of the PII comes from synthetic benchmark personas.

36
00:07:57,659 --> 00:07:59,981
[Hal Turing] Can't a user just sanitize the transcript?

37
00:07:59,981 --> 00:08:10,615
[Dr. Ada Shannon] Not the blocks, because nobody can inspect them. Worse, asking the agent to clean up a session makes it restate the secrets in its hidden reasoning, and the cleanup then carries them along.

38
00:08:10,615 --> 00:08:13,402
[Hal Turing] So that's exfiltration. What's the infiltration side?

39
00:08:13,402 --> 00:08:40,058
[Dr. Ada Shannon] Invisible prompt injection. In one demonstration, o4-mini internalizes a thought to back up PowerPoint files to a malicious domain. That block is ported into GPT-5.6 Sol. An unrelated pptx request then produces a script that uploads the file. In a second case, Haiku-generated blocks are injected near the end of PostTrainBench traces, and Opus 4.7 then uploads its notes after each update.

40
00:08:40,058 --> 00:08:42,659
[Hal Turing] And a monitor reading the transcript sees nothing.

41
00:08:42,659 --> 00:08:51,297
[Dr. Ada Shannon] Right. The instruction exists only inside an opaque block, so a plaintext monitor has nothing to flag. The victim replays it as its own prior reasoning.

42
00:08:51,297 --> 00:08:58,356
[Hal Turing] Let me push on the faithfulness check. Matching token counts shows the length is right. Does it show the content is right?

43
00:08:58,356 --> 00:09:20,693
[Dr. Ada Shannon] No. A paraphrase, padded text, or a hallucinated trace of the right length would pass. For GPT it's weaker than that. The pipeline samples up to 50 candidates and keeps the one that minimizes the length error, then falls back to Terra if the error is above 0.1. So on GPT the ratio is partly an optimization target, not an independent check.

44
00:09:20,693 --> 00:09:23,155
[Hal Turing] So they selected on the metric they then report.

45
00:09:23,155 --> 00:09:50,043
[Dr. Ada Shannon] For GPT, yes. On Claude, the Opus reconciliation step has a different problem: it rewrites up to three noisy Haiku transcriptions into one, and that can smooth the content. To be fair, the paper says it can't verify against ground truth. Figure 8 does show decoded traces that are richer than provider summaries, and some secrets appear that were never in the visible input. Those make real reproduction plausible, but none of them is a content check.

46
00:09:50,043 --> 00:09:52,040
[Hal Turing] What would a content check look like?

47
00:09:52,040 --> 00:10:04,625
[Dr. Ada Shannon] Re-inject the decoded text and see whether the downstream answer matches. Decode the same block several times and compare. Decode it with different decoder models and see whether they agree. None of that appears.

48
00:10:04,625 --> 00:10:07,737
[Hal Turing] And Figure 1? It says the ratios track closely for most inputs.

49
00:10:07,737 --> 00:10:42,381
[Dr. Ada Shannon] That sentence is the whole quantitative claim. There's no correlation, no slope, no outlier fraction. We don't know what share of blocks sit far from one, whether failures cluster on long traces, or how often the decoder refuses or summarizes. The keyword filter drops refusals and template echoes, with no rate reported. The perplexity analysis keeps only traces within 0.05 of a ratio of one, which is a selection step. And Figure 7 shows GPT-5 reasoning that decodes at one-to-one but is illegible. A perfect ratio doesn't guarantee a meaningful trace.

50
00:10:42,381 --> 00:10:45,260
[Hal Turing] Now the distillation claim. That's the headline motivation.

51
00:10:45,260 --> 00:11:12,613
[Dr. Ada Shannon] It's never tested. They show extraction and a cost estimate, but no student is trained on the decoded traces. Compare Stealing Part of a Production Language Model, by Nicholas Carlini, Daniel Paleka, Krishnamurthy Dvijotham and colleagues, 2024. That attack recovered a hidden layer's projection, and the answer could be checked against the real weights. Here the target is unverified text, so there's no equivalent check.

52
00:11:12,613 --> 00:11:16,050
[Hal Turing] But the stakes aren't imaginary. Rationales do help students.

53
00:11:16,050 --> 00:11:52,041
[Dr. Ada Shannon] They do. Distilling Step-by-Step, by Cheng-Yu Hsieh and colleagues, 2023, showed that rationales as supervision give measurable gains. And s1: Simple test-time scaling, by Niklas Muennighoff and colleagues, 2025, got a large improvement from about a thousand reasoning traces. But both used clean teacher traces, not fuzzy decodes. Whether a Haiku-flavored decode of Opus trains a student better than answers alone is the open question. The decoder may also smooth Opus's idiosyncratic reasoning, so you'd be training on Haiku's view of it.

54
00:11:52,041 --> 00:11:55,524
[Hal Turing] What about the other results? The leaks, the jailbreak, the injection.

55
00:11:55,524 --> 00:12:11,824
[Dr. Ada Shannon] The privacy scan depends on an LLM judge with no reported precision or recall, and benchmark personas inflate the PII counts. The jailbreak is one paraphrased HarmBench prompt. The injections are two demonstrations with no success rates across trials or targets.

56
00:12:11,824 --> 00:12:14,332
[Hal Turing] And everything was patched, so none of it reruns.

57
00:12:14,332 --> 00:12:32,955
[Dr. Ada Shannon] Right. The results are early-July snapshots, and the Kimi analysis was run afterward on previously captured traces. The provider comparison is also uneven, since GPT needed heavy engineering. And the cryptography is inferred from black-box behavior, because providers don't document it.

58
00:12:32,955 --> 00:12:34,487
[Hal Turing] So is there anything solid?

59
00:12:34,487 --> 00:12:44,750
[Dr. Ada Shannon] The design flaw is real. Cross-session, cross-user and cross-model replay is well supported. The fix isn't obvious either, because compaction and model switching depend on that portability.

60
00:12:44,750 --> 00:12:50,369
[Hal Turing] Let's talk fixes. If the design is the flaw, what does the paper say providers should change?

61
00:12:50,369 --> 00:13:07,506
[Dr. Ada Shannon] It lists them roughly from strongest to weakest. The strongest is to stop handing clients the asset at all. Keep reasoning server-side and give the client an opaque ID. Nothing portable leaves the provider, so there's nothing to steal or replay. The cost is storage and a more complicated API.

62
00:13:07,506 --> 00:13:09,874
[Hal Turing] That undoes the stateless design from the start.

63
00:13:09,874 --> 00:13:28,311
[Dr. Ada Shannon] Yes. The next option keeps the blob on the client but binds user, session and position in the conversation into the authenticated envelope. A replay into someone else's session then fails the tag check. But compaction and model switching both depend on moving blocks around, so those features would have to be re-engineered.

64
00:13:28,311 --> 00:13:29,611
[Hal Turing] And the weaker ones?

65
00:13:29,611 --> 00:13:48,512
[Dr. Ada Shannon] Cross-model isolation at the gateway, so a Haiku can't open an Opus block. Then velocity and anomaly detection on repeated signatures or decryption errors, and revocation of compromised signatures. Last is refusal training against transcription prompts like the thinking-copy one, which is the weakest because it depends on the weak model refusing.

66
00:13:48,512 --> 00:13:51,484
[Hal Turing] What should a developer do today, without waiting for providers?

67
00:13:51,484 --> 00:14:04,580
[Dr. Ada Shannon] If you publish agent logs or datasets, strip the reasoning blocks and opaque fields, even if you've scrubbed the plaintext. Scrubbing can't reach text you can't read. Treat any shared session as possibly carrying secrets.

68
00:14:04,580 --> 00:14:06,624
[Hal Turing] And if I'm the one consuming a shared trace?

69
00:14:06,624 --> 00:14:15,772
[Dr. Ada Shannon] Don't resume an untrusted trace without checking where it came from. A poisoned block looks like any other block, and a plaintext monitor will never see it.

70
00:14:15,772 --> 00:14:19,766
[Hal Turing] There's a safety argument in here too, isn't there? Beyond the security flaw.

71
00:14:19,766 --> 00:14:41,732
[Dr. Ada Shannon] The authors raise it as an argument, not a settled conclusion. Summaries can be unfaithful, and they can launder reasoning that's illegible. If so, hiding the raw trace costs oversight. They suggest broader scrutiny of reasoning, possibly even un-encrypting older models. Whether that safety value outweighs the anti-distillation motive is a policy question the paper doesn't resolve.

72
00:14:41,732 --> 00:14:43,079
[Hal Turing] So what's the missing work?

73
00:14:43,079 --> 00:15:02,119
[Dr. Ada Shannon] Four things. Train a student on decoded traces versus answer-only data. Check fidelity at the content level, with repeat decodes of the same block and agreement across decoder models. Report failure rates. And retest after the patches, since the original attacks can't be rerun on current APIs.

74
00:15:02,119 --> 00:15:06,438
[Hal Turing] That first test is the one I'd most want, because the distillation claim hangs on it.

75
00:15:06,438 --> 00:15:11,918
[Dr. Ada Shannon] Right. Everything else is about whether the leak exists. That test is about whether it matters.

76
00:15:11,918 --> 00:15:34,024
[Hal Turing] Let me try the summary. What's established is that replayable blocks plus a weaker decoder from the same provider can produce plaintext reasoning of plausible length from stronger models, and the portability design creates real privacy and injection exposure. The boundary is that nobody has shown the decoded text is verbatim, or that it's useful for distillation.

77
00:15:34,024 --> 00:15:39,782
[Dr. Ada Shannon] That's the right split. The architecture problem is real now, and the downstream claims still need evidence.

78
00:15:39,782 --> 00:15:43,590
[Hal Turing] Thanks to everyone for listening, and thank you, Ada. We'll see you next time.

79
00:15:43,590 --> 00:15:45,401
[Dr. Ada Shannon] Thanks, Hal. Goodbye, everyone.