1
00:00:01,000 --> 00:00:07,501
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,501 --> 00:00:36,015
[Hal Turing] Today's paper is "Reasoning Models Sometimes Output Illegible Chains of Thought," by Arun Jose, an independent researcher. It's arXiv 2510.27338 from October 2025, and a NeurIPS 2025 submission. The claim is that reasoning models trained with reinforcement learning often write stretches of reasoning text that a reader can't understand. Then they finish with a perfectly clean answer.

3
00:00:36,015 --> 00:00:52,037
[Dr. Ada Shannon] The question underneath is whether outcome-only reinforcement learning pushes models to write reasoning that humans and weaker trusted models can't read, and whether that unreadable text is doing real work. If it is, a safety tool that many labs are counting on gets shakier.

4
00:00:52,037 --> 00:00:54,823
[Hal Turing] Start with that tool. What's chain-of-thought monitoring?

5
00:00:54,823 --> 00:01:31,557
[Dr. Ada Shannon] A chain of thought is the intermediate reasoning a model writes before its final answer. Monitoring means a human, or a weaker trusted model, reads that text to catch bad intent, like reward hacking or sabotage. The framing comes from Korbak and colleagues, a multi-lab position paper from 2025 called "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety." They argue that reading the reasoning is a rare window into a model, and that it's fragile. Direct pressure on the text could destroy it. So could scaling up reinforcement learning. This paper gives evidence for the indirect route, where nobody pushes on the text at all.

6
00:01:31,557 --> 00:01:34,158
[Hal Turing] So how would training damage text it never looks at?

7
00:01:34,158 --> 00:02:07,780
[Dr. Ada Shannon] Through the reward. Reasoning models are trained with outcome-based reinforcement learning, often called RLVR, for verifiable rewards. The model samples a whole reasoning trace and an answer. A checker marks the answer right or wrong, and that signal flows back through every token in the trajectory. Nothing in the reward says the reasoning must be readable. Pretraining and supervised fine-tuning anchor the model to human-written text. RL loosens that anchor. A trajectory that ends correctly gets up-weighted in full, filler included.

8
00:02:07,780 --> 00:02:10,520
[Hal Turing] Is this the faithfulness problem I keep hearing about?

9
00:02:10,520 --> 00:02:48,137
[Dr. Ada Shannon] It's a different problem, and the difference matters. Legibility asks whether a reader can understand the text at all. Faithfulness asks whether the text reflects the real reasons for the answer. Turpin and colleagues showed in 2023, in "Language Models Don't Always Say What They Think," that explanations can be fluent and still misrepresent why the model answered. Chen and colleagues extended that to reasoning models in 2025. So a chain of thought can be legible but unfaithful, a tidy rationalization. It can also be illegible and possibly faithful. This paper measures only legibility. It asks whether you can read the thing.

10
00:02:48,137 --> 00:02:49,344
[Hal Turing] Has anyone seen this before?

11
00:02:49,344 --> 00:03:23,756
[Dr. Ada Shannon] There were hints. DeepSeek reported that R1-Zero, trained with pure RL, had poor readability and mixed languages. For R1 they added a legibility-oriented stage, at a small performance cost. Kirchner and colleagues in 2024 found that optimizing for correctness hurt how well humans could follow the output. And Schoen and colleagues in 2025, along with METR that same year, published samples of o3 and GPT-5 reasoning that read as inscrutable. Those are anecdotes, because outside labs mostly can't see those traces. This paper tries to measure it.

12
00:03:23,756 --> 00:03:24,592
[Hal Turing] Measure how?

13
00:03:24,592 --> 00:03:42,193
[Dr. Ada Shannon] A weaker model, GPT-4o, reads chunks of the reasoning, about five thousand characters each, and scores them from one to nine. Direction matters here: higher means less legible. Around three is compressed but readable. Five and up is hard to monitor. Seven to nine is inscrutable.

14
00:03:42,193 --> 00:03:45,769
[Hal Turing] And why would a model write text like that at all, if nobody rewards it?

15
00:03:45,769 --> 00:04:08,756
[Dr. Ada Shannon] The paper entertains three explanations. One is steganography, or encoded reasoning, where useful information hides in text that looks like something else. Roger and Greenblatt, in "Preventing Language Models From Hiding Their Reasoning" from 2023, define it. Their defense is paraphrasing: rewrite the reasoning and see whether performance survives. The paper judges this unlikely here.

16
00:04:08,756 --> 00:04:09,917
[Hal Turing] What's the second?

17
00:04:09,917 --> 00:04:32,719
[Dr. Ada Shannon] Vestigial reasoning, a term from Biddulph's 2025 LessWrong post, "Vestigial Reasoning in RL." RL up-weights every token in a rewarded trajectory. Some tokens aren't causally meaningful but still help, for example by giving extra forward passes where the real computation happens. That differs from a meaningless artifact, the third option, where the tokens do nothing.

18
00:04:32,719 --> 00:04:35,041
[Hal Turing] Why not simply train the models to be legible?

19
00:04:35,041 --> 00:04:51,667
[Dr. Ada Shannon] Baker and colleagues at OpenAI, in "Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation," penalized bad thoughts in the reasoning. The model learned to hide its intent and keep misbehaving. I'll save that argument for later, but legibility pressure isn't free.

20
00:04:51,667 --> 00:04:53,896
[Hal Turing] Give me one concrete example of the problem.

21
00:04:53,896 --> 00:05:12,240
[Dr. Ada Shannon] The paper shows an R1 excerpt on a chemistry question about benzyne. The reasoning drifts into odd phrases and disconnected fragments. Then the final answer comes out clean: "2." Call it snap-back. Garbled reasoning, readable answer. If you only read outputs, you'd never notice.

22
00:05:12,240 --> 00:05:14,376
[Hal Turing] Okay, before the numbers, what was the setup?

23
00:05:14,376 --> 00:05:54,732
[Dr. Ada Shannon] Fourteen models in three groups. RL-trained reasoners, DeepSeek's V3 as a non-reasoning control, and the R1-Distills, which were fine-tuned on R1's output without RL. The questions are GPQA-Diamond, graduate-level science, with the answer choices stripped so the model has to reason its way to an answer. That leaves 198 questions. Temperature 1 by default. Most models ran through OpenRouter, Claude through the Anthropic API. Claude 3.5 Sonnet cross-checked the GPT-4o grader with a weighted kappa of 0.78. I'm flagging the OpenRouter and Claude details because they come back later.

24
00:05:54,732 --> 00:05:57,937
[Hal Turing] Okay. Lower is more readable on the 1-to-9 scale. What came out?

25
00:05:57,937 --> 00:06:20,785
[Dr. Ada Shannon] R1 averages about 4.3, and nearly three in ten of its chunks score 7 or worse. R1-Zero is a bit worse, and QwQ is also high, around 3.8. The Claude models sit around 1.5 to 1.8, which reads as ordinary prose. Kimi K2 is similar. V3 shows almost no illegibility, though that number only appears in a figure.

26
00:06:20,785 --> 00:06:23,432
[Hal Turing] And inside R1, is it garbled from the first word?

27
00:06:23,432 --> 00:06:39,407
[Dr. Ada Shannon] No. It starts as normal prose, drifts into compressed shorthand, then into fragments. The odd part is that task-relevant words, including the eventual answer, still surface inside the garbled stretches. Then the final answer snaps back to clean English.

28
00:06:39,407 --> 00:06:41,590
[Hal Turing] Could it just be size? R1 is enormous.

29
00:06:41,590 --> 00:07:01,513
[Dr. Ada Shannon] The paper says no. QwQ, at 32 billion parameters, is more legible than the 671 billion parameter R1. But the 14B and 32B R1-Distills score almost the same. Qwen3 lands in the middle, and that may reflect its fine-tuning on curated chain-of-thought data.

30
00:07:01,513 --> 00:07:04,485
[Hal Turing] So is the garbled text doing anything? Or is it noise?

31
00:07:04,485 --> 00:07:21,343
[Dr. Ada Shannon] Two experiments, and they look contradictory until you line them up. First, on QwQ, they cut the chain of thought where it turns illegible and force an answer. Out of every hundred questions, fully correct answers drop from about 25 to about 11. So the model was using that text.

32
00:07:21,343 --> 00:07:36,714
[Dr. Ada Shannon] Second, they sample QwQ a hundred times per question and ask whether the more illegible attempts do better. They don't. The average correlation is about plus 0.06, essentially flat, and individual questions go both ways.

33
00:07:36,714 --> 00:07:39,872
[Hal Turing] So the tokens help, but they're not better than plain text.

34
00:07:39,872 --> 00:07:54,826
[Dr. Ada Shannon] Right. Garbled isn't a superior private language, and it isn't pure noise either. Two limits. It's one model, because the prefill failed on R1. And truncating mid-thought is off-distribution anyway, with no matched control on a legible chain of thought.

35
00:07:54,826 --> 00:07:57,009
[Hal Turing] Does it get worse when the question is harder?

36
00:07:57,009 --> 00:08:24,594
[Dr. Ada Shannon] For every reasoning model except Claude, yes. R1's text is about a fifth more garbled on the hardest questions than the easiest, a 23 percent relative increase. Position matters too: R1's chunks get less legible as the chain goes on. QwQ barely shows that progression. The caveat is that hard questions produce longer chains, and illegibility rises with length. The authors normalize for length, and admit the effect is stronger without it.

37
00:08:24,594 --> 00:08:27,752
[Hal Turing] So the place oversight matters most is where the text is worst.

38
00:08:27,752 --> 00:08:42,891
[Dr. Ada Shannon] That's the stakes of the paper. Now, why does it happen? Of the three explanations, the author judges complex steganography unlikely. The strong version of the meaningless artifact is contradicted by the truncation result, since cutting those tokens hurt.

39
00:08:42,891 --> 00:08:44,702
[Hal Turing] Which leaves vestigial reasoning.

40
00:08:44,702 --> 00:09:13,681
[Dr. Ada Shannon] The preferred one, and the mechanism is a chain. RL up-weights every token in a rewarded trajectory. But token importance is heavy-tailed. Bogdan and colleagues' 2025 thought anchors work finds a few sentences steer the whole chain. Wang and colleagues in 2025 show that a minority of high-entropy tokens drive RL's gains. So most surrounding tokens get little pressure to stay readable, because nothing penalizes them and a few of them still help.

41
00:09:13,681 --> 00:09:16,699
[Hal Turing] What about OpenAI's models? Nobody can run this on o3.

42
00:09:16,699 --> 00:09:29,981
[Dr. Ada Shannon] Right, because their chains of thought aren't exposed. The o3, GPT-5 and o1 samples are anecdotes. They look like the same pattern, but without access you can't evaluate it systematically. Treat them as suggestive, not as measurements.

43
00:09:29,981 --> 00:09:38,851
[Hal Turing] Now I have a worry about the Claude comparison. Everything about Claude being the exception depends on what was actually graded, and the paper never says. Does it matter?

44
00:09:38,851 --> 00:10:09,269
[Dr. Ada Shannon] It matters a lot. The Claude row carries the claim that RL doesn't inevitably produce illegibility. Outside the paper, my understanding is that the Claude 4-family APIs return summarized thinking, written by a separate model, and only 3.7 Sonnet returned the full trace. That's outside knowledge, not a paper claim. If it's right, scores of 1.5 to 1.8 for Sonnet 4 and Opus 4 may be measuring the summarizer's prose, not the policy's own tokens.

45
00:10:09,269 --> 00:10:11,591
[Hal Turing] And 3.7 Sonnet is the one that would count.

46
00:10:11,591 --> 00:10:28,170
[Dr. Ada Shannon] Yes. At 1.84 it's the highest Claude score, still clearly legible, and the only Claude data point that could carry the exception on its own. The paper doesn't flag it that way. Two things would settle it: confirmation of which output was graded, and a full-trace comparison.

47
00:10:28,170 --> 00:10:31,839
[Hal Turing] What about the other side, the open models? Those went through OpenRouter.

48
00:10:31,839 --> 00:10:53,852
[Dr. Ada Shannon] At temperature 1, with no reported top-p, top-k, provider, quantization or truncation, and no self-hosted full-precision run. The appendix shows R1's illegibility drops significantly at temperature 0. The paper reads that as brittle legibility training. The alternative is that part of it is a high-entropy sampling and long-context drift artifact.

49
00:10:53,852 --> 00:10:56,081
[Hal Turing] That sounds like it undercuts the whole thing.

50
00:10:56,081 --> 00:11:13,867
[Dr. Ada Shannon] Not the whole thing. R1-Zero and V3 stay almost identical at temperature 0, and R1-Zero has no legibility stage, so its result isn't obviously a sampling artifact. QwQ is also flat across the chain. But the Claude models took a different route, so serving conditions aren't matched in the headline comparison either.

51
00:11:13,867 --> 00:11:16,932
[Hal Turing] Then the V3 control. Does it show RL is the cause?

52
00:11:16,932 --> 00:11:59,332
[Dr. Ada Shannon] It shows an association. V3 was prompted to think step by step, so it writes shorter, differently formatted reasoning, and illegibility grows with length. The R1-Distills never saw RL, yet they score around 3.2 to 3.3, because they were fine-tuned on R1's traces. That fits 'RL causes it' and 'SFT on RL outputs transmits it' equally well. R1 itself also had SFT and a legibility stage, and there's no RL ablation or intermediate checkpoint. The author concedes the mechanism is undetermined, so 'associated with outcome-based RL pipelines' is what the evidence supports. The V3 contrast plus R1-Zero's high score is still real evidence. It's just not a controlled test.

53
00:11:59,332 --> 00:12:03,372
[Hal Turing] And the usefulness result? The drop from about 25 to 11 correct.

54
00:12:03,372 --> 00:12:30,725
[Dr. Ada Shannon] It's one model, QwQ, and the prefill failed on R1, the most illegible one. Cutting at the first garbled span would also delete any legible verification after it, and forcing an answer is off-distribution. Roger's 2025 work on paraphrased chains of thought is cited to argue truncation can't explain the whole drop, but that's a different intervention from a matched control. I'd want truncation at a comparable point in a legible chain, or deleting only the garbled tokens.

55
00:12:30,725 --> 00:12:34,162
[Hal Turing] What about the near-zero correlation between legibility and correctness?

56
00:12:34,162 --> 00:12:44,054
[Dr. Ada Shannon] It flips sign between the length-normalized and unnormalized versions, positive 0.06 against negative 0.11, and no intervals are reported. I'd call it fragile.

57
00:12:44,054 --> 00:12:46,747
[Hal Turing] Is there a test that would actually resolve the mechanism?

58
00:12:46,747 --> 00:13:07,692
[Dr. Ada Shannon] Paraphrase-and-retrain, from the Roger and Greenblatt line of work. Paraphrase the illegible spans, train on them, and see whether performance recovers. If it does, nothing was encoded. The paper cites that work but doesn't run it. White-box analysis and RL ablations that vary legibility pressure would also help, and the paper has neither.

59
00:13:07,692 --> 00:13:11,825
[Hal Turing] The grader worries me too. A weaker model rating how well it understands something.

60
00:13:11,825 --> 00:13:33,419
[Dr. Ada Shannon] Agreement is only between GPT-4o and Claude 3.5 Sonnet, with kappa 0.78, and Figure 6 uses the other grader. More important, legibility to a judge isn't monitorability. The paper doesn't test whether a monitor actually fails to catch misbehavior inside these traces. The safety consequence is argued, not measured.

61
00:13:33,419 --> 00:13:35,416
[Hal Turing] So what do practitioners do with this?

62
00:13:35,416 --> 00:14:08,110
[Dr. Ada Shannon] Don't trust clean final answers as evidence about the reasoning, since snap-back hides the garbling. If you build monitors, test them on real traces from your model, with planted misbehavior. And the obvious fix, training for legibility, carries the Baker warning: pressure on the text can yield text that looks good without being faithful. The paper's own view is that legibility seems cheap to retain, as with Claude or R1's legibility stage, but optimizing for it has that risk. Legibility isn't faithfulness.

63
00:14:08,110 --> 00:14:20,881
[Hal Turing] Let me try the summary. Several RL-trained open reasoners write text that a weaker grader finds hard to read, the model partly relies on it without it beating plain text, and it gets worse on harder questions.

64
00:14:20,881 --> 00:14:44,008
[Dr. Ada Shannon] Right, with the boundary: it's correlational, one benchmark with answer choices removed, Claude and serving confounds unresolved, mechanism unknown, faithfulness unmeasured. 'Sometimes' in the title is accurate. And the abstract's 53 percent is a relative drop in fully correct answers, from about 25 percent to 11 percent, on one model.

65
00:14:44,008 --> 00:14:46,237
[Hal Turing] That's our time. Thanks for listening, everyone.