1
00:00:01,000 --> 00:00:07,501
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,501 --> 00:00:34,761
[Hal Turing] Today's paper is "Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought," from Keshav Ramji, Tahira Naseem, and Ramón Fernandez Astudillo at IBM Research AI. It's an arXiv paper from April 2026. The question is whether a model can swap a long written-out chain of thought for a short string of brand-new tokens that don't mean anything in English, and still answer as accurately.

3
00:00:34,761 --> 00:00:47,068
[Dr. Ada Shannon] And the authors say they do it with post-training alone. No new pre-training run and no change to the architecture. That's the claim we'll be checking. First, though, we need to explain why anyone wants to shorten chain of thought at all.

4
00:00:47,068 --> 00:00:54,545
[Hal Turing] Right. Define the baseline for us. Verbal chain of thought is the model writing out its reasoning in plain language before it answers.

5
00:00:54,545 --> 00:01:17,300
[Dr. Ada Shannon] Yes. It's accurate, but you pay for every token in latency and compute. Reinforcement learning makes it worse, because rewarding correct answers tends to lengthen the reasoning. In this paper, a Qwen3-8B model trained with supervised fine-tuning and then RL averages about 1,671 tokens per MATH-500 problem. That's the cost they're attacking.

6
00:01:17,300 --> 00:01:21,387
[Hal Turing] And there's a second complaint, about whether the written reasoning is even real.

7
00:01:21,387 --> 00:01:42,424
[Dr. Ada Shannon] Right. Lanham and colleagues at Anthropic, in 2023, measured how much answers actually depend on the stated chain of thought. Turpin and colleagues at NYU, also 2023, showed that models can give plausible explanations that leave out what actually drove the answer. So the words are expensive, and they may not be faithful either.

8
00:01:42,424 --> 00:01:47,161
[Hal Turing] Which makes dropping the words tempting. So how did people try that before this paper?

9
00:01:47,161 --> 00:02:13,771
[Dr. Ada Shannon] Start with the simplest idea, pause or filler tokens. Goyal and colleagues, out of Google in 2024, in "Think before you speak: Training Language Models with Pause Tokens," added semantically empty tokens so the model gets extra forward-pass positions to compute in. The catch is that it works mainly when you pre-train with them. Fine-tune them in afterward and the gains tend to fail. This paper reproduces that failure.

10
00:02:13,771 --> 00:02:16,929
[Hal Turing] Then there's the line of work that tries to internalize the reasoning.

11
00:02:16,929 --> 00:02:42,610
[Dr. Ada Shannon] Yes. Deng and colleagues in 2024, "From explicit CoT to implicit CoT," introduced Stepwise Internalization. You start with explicit reasoning and gradually remove steps during fine-tuning, until the model answers directly. Cheng and Van Durme at Johns Hopkins, also 2024, did Compressed Chain of Thought, which trains the model to produce a short run of dense continuous vectors that stand in for the verbal steps.

12
00:02:42,610 --> 00:02:46,418
[Hal Turing] Continuous vectors. That's what people mean by latent reasoning, right?

13
00:02:46,418 --> 00:03:16,790
[Dr. Ada Shannon] Latent reasoning is intermediate computation that never appears as readable language. Coconut, by Hao and colleagues at Meta, "Training Large Language Models to Reason in a Continuous Latent Space," feeds the last hidden state back in as the next input. The model thinks in vectors, not words. CODI, from Shen and colleagues in 2025, compresses reasoning into continuous latents using self-distillation. It aligns the hidden states of the compressed pass with those of an explicit chain of thought.

14
00:03:16,790 --> 00:03:21,063
[Hal Turing] Self-distillation meaning the model teaches itself. No separate bigger teacher.

15
00:03:21,063 --> 00:03:42,704
[Dr. Ada Shannon] Yes. The same network plays both roles. We'll get to how this paper uses it. One more reference, on the length side. Aggarwal and Welleck at Carnegie Mellon, 2025, wrote L1, which uses reinforcement learning to make the model obey a requested reasoning length. That's a different route to short reasoning, and it matters later.

16
00:03:42,704 --> 00:03:45,211
[Hal Turing] So where does Abstract-CoT sit among all those?

17
00:03:45,211 --> 00:04:07,967
[Dr. Ada Shannon] Between the two camps. It stays discrete, like ordinary language-model output. But the tokens aren't words. The authors reserve a new codebook of 64 abstract tokens, named things like TOKEN_A and TOKEN_B. The model emits up to 128 of them between a begin-abstract marker and an end-abstract marker, and only then writes the answer.

18
00:04:07,967 --> 00:04:11,543
[Hal Turing] Those are brand-new vocabulary entries. What do they mean at the start?

19
00:04:11,543 --> 00:04:20,506
[Dr. Ada Shannon] Nothing. That's the cold-start problem. Their embeddings are random, so the model has no reason to treat them as anything but noise. Training has to give them a use.

20
00:04:20,506 --> 00:04:26,357
[Hal Turing] And that's the failure with pause tokens. Empty positions, nothing forcing them to carry content.

21
00:04:26,357 --> 00:04:41,915
[Dr. Ada Shannon] Right. Here the authors add a forcing structure during warm-up. An attention mask hides the verbal chain of thought from the answer tokens. The answer sees only the prompt and the abstract tokens, so whatever it needs from the reasoning has to pass through those abstract positions.

22
00:04:41,915 --> 00:04:43,865
[Hal Turing] They call that an information bottleneck.

23
00:04:43,865 --> 00:05:03,695
[Dr. Ada Shannon] They do, and the term needs care. The data-processing inequality says the answer can't receive more information about the reasoning than the abstract positions hold. But that's an argument about structure. They don't measure or optimize mutual information anywhere. It's a training-time constraint with an information-theoretic motivation.

24
00:05:03,695 --> 00:05:15,119
[Hal Turing] Fair. So the pitch is discrete tokens, a bounded length of 128, and a bottleneck that makes the tokens carry reasoning, all added after the base model exists.

25
00:05:15,119 --> 00:05:30,258
[Dr. Ada Shannon] That's the positioning. Against Coconut and CODI, you keep ordinary discrete decoding, so the machinery is standard token sampling. Against pause tokens, you don't need pre-training. Whether that's better is a separate question, and we'll test it against the evidence.

26
00:05:30,258 --> 00:05:36,574
[Hal Turing] So the plan: next, how the two-stage recipe works, then what the experiments show, and then what they leave unanswered.

27
00:05:36,574 --> 00:05:44,469
[Hal Turing] Let's get into the recipe. Stage one is a warm-up loop, and I want the causal chain. What goes in, and what does each step change?

28
00:05:44,469 --> 00:06:11,358
[Dr. Ada Shannon] Start with the problem: the new tokens are meaningless, so the first job is forcing them to carry something. In iteration one, each verbal reasoning step gets a random handful of abstract tokens, between one and half the step's length, drawn uniformly from the 64-token codebook. Then they train with the mask. The abstract tokens can see the prompt and the verbal chain of thought. The answer can see only the prompt and the abstract tokens.

29
00:06:11,358 --> 00:06:15,723
[Hal Turing] So the only road from the reasoning to the answer runs through those abstract positions.

30
00:06:15,723 --> 00:06:27,565
[Dr. Ada Shannon] Yes. The gradient pushes the abstract hidden states to hold whatever the answer needs, and the paper says the cap m_max sets the channel capacity. As before, that's a design argument, not a measurement.

31
00:06:27,565 --> 00:06:30,677
[Hal Turing] But at test time there's no verbal chain of thought to peek at.

32
00:06:30,677 --> 00:06:53,897
[Dr. Ada Shannon] Right, and that's what self-distillation fixes. The model generates abstract sequences from the prompt alone, then trains on prompt, its own abstract trace, and the gold answer. The teacher is gone. The model learns to write the trace it would have needed. Iterations two and three use the model's own traces in the bottleneck step instead of random ones. The loop runs three times.

33
00:06:53,897 --> 00:06:55,011
[Hal Turing] And the length cap?

34
00:06:55,011 --> 00:07:13,309
[Dr. Ada Shannon] Constrained decoding. During the abstract segment, the probabilities are masked to the codebook plus the end delimiter, then renormalized. At 128 tokens the end delimiter is forced. After that, the answer is decoded freely. So the bound applies only to the abstract segment, not the answer.

35
00:07:13,309 --> 00:07:15,166
[Hal Turing] Stage two is the reinforcement learning?

36
00:07:15,166 --> 00:07:34,625
[Dr. Ada Shannon] GRPO, sampling a group of abstract-plus-answer pairs per prompt. A generative reward model, gpt-oss-20b, scores them, which lets it handle non-verifiable tasks like instruction following. The policy gradient covers both abstract and answer tokens, with a KL penalty back to the warm-up model.

37
00:07:34,625 --> 00:07:35,739
[Hal Turing] What's the setup?

38
00:07:35,739 --> 00:07:53,015
[Dr. Ada Shannon] Qwen3-8B, Qwen3-4B, and Granite-4.0-Micro at 3 billion parameters. About 600,000 Dolci-Think samples for warm-up, Dolci-Think prompts for RL, a million RL episodes, and 8 to 32 H100s. Baselines are evaluated with thinking mode off.

39
00:07:53,015 --> 00:07:55,290
[Hal Turing] Then the headline. MATH-500, Qwen3-8B.

40
00:07:55,290 --> 00:08:15,770
[Dr. Ada Shannon] MATH-500 is competition-style math, and higher accuracy is better. Verbal SFT plus RL gets about 93 of 100 right and writes roughly 1,700 tokens per problem. Abstract-CoT gets about 91 and writes 144. Roughly two fewer correct answers per hundred, for one-twelfth of the output.

41
00:08:15,770 --> 00:08:19,068
[Hal Turing] Does that 144 include the answer, or just the abstract trace?

42
00:08:19,068 --> 00:08:28,356
[Dr. Ada Shannon] Good thing to pin down. The Table 1 caption says the column combines reasoning and response tokens. So 144 is everything generated.

43
00:08:28,356 --> 00:08:29,377
[Hal Turing] And the other benchmarks?

44
00:08:29,377 --> 00:08:57,148
[Dr. Ada Shannon] On AlpacaEval, where a judge picks which answer it prefers, Abstract-CoT is ahead by about two and a half points while writing under half the tokens. On HotpotQA, multi-hop question answering, the two are roughly tied, with a bit over four times fewer tokens. The pattern holds on the other two models. The 4B is about a point and a half behind on MATH-500, and Granite is slightly ahead. These are single runs with no variance reported, so I wouldn't read the small gaps as wins.

45
00:08:57,148 --> 00:09:00,120
[Hal Turing] Now the ablations. Is the warm-up pulling its weight?

46
00:09:00,120 --> 00:09:21,576
[Dr. Ada Shannon] The ladder is clean. RL with no warm-up scores 82, no better than the untouched model's 82.4. Warm-up alone reaches 88, above the 85.8 you get from plain fine-tuning with no reasoning at all, but below verbal. Only warm-up plus RL gets to 90.8. The paper's reading is that the warm-up gives RL something to work with.

47
00:09:21,576 --> 00:09:24,826
[Hal Turing] And the pause tokens? Same token count, no warm-up.

48
00:09:24,826 --> 00:09:49,393
[Dr. Ada Shannon] At about 140 tokens they score 78.6, below the untouched model. The full recipe at the same budget scores about 91. That's twelve more correct answers per hundred. But they fine-tuned pause tokens in without pre-training them, the case Goyal and colleagues already said fails. So it shows the extra positions alone don't help. It doesn't show the abstract vocabulary beats every alternative.

49
00:09:49,393 --> 00:09:54,827
[Hal Turing] What about Stepwise Internalization, the method that gradually strips the reasoning steps out?

50
00:09:54,827 --> 00:10:11,359
[Dr. Ada Shannon] That's the tougher comparison. It reaches 88.6 at 169 tokens, about two points under the full recipe. The paper says it takes 22 training iterations versus three warm-up rounds plus RL here, but that compares iteration counts, not total compute, so I wouldn't lean on it.

51
00:10:11,359 --> 00:10:14,796
[Hal Turing] Last item. The paper says the token usage follows a power law?

52
00:10:14,796 --> 00:10:41,267
[Dr. Ada Shannon] Initial usage is uniform and random. By the end of RL, a few tokens dominate and a long tail barely appears, Zipf-like, with TOKEN_F pulling far ahead. The authors read that as a learned reasoning language. What it shows is reuse. A skewed distribution can come from a handful of useful habits, or from a default token the model leans on. Without interpretability work, it doesn't tell us what any token means.

53
00:10:41,267 --> 00:10:48,883
[Hal Turing] Or a degenerate policy that found a cheap habit. So before anyone calls it a language, what is the biggest thing the headline number leaves out?

54
00:10:48,883 --> 00:11:22,459
[Dr. Ada Shannon] The denominator. The 11.6x compares 1,671 tokens from SFT plus RL, a baseline RL had already inflated, against 144 for Abstract-CoT. Both are totals, reasoning plus response. The text defines the ratio on reasoning alone, but the quoted ratios match the totals exactly. And with a 128-token cap, a 144 total leaves room for only about 16 response tokens if the trace used its full length. The paper never reports the trace length on its own, so we don't know the split.

55
00:11:22,459 --> 00:11:25,710
[Hal Turing] And the no-CoT SFT model sits at 394 tokens.

56
00:11:25,710 --> 00:11:40,942
[Dr. Ada Shannon] Yes, at 85.8 on MATH-500. So a fairer claim is about five more correct answers per hundred than no-CoT SFT, for roughly a third of that model's output. That's less dramatic than twelve-fold, and it's still a real result.

57
00:11:40,942 --> 00:11:45,725
[Hal Turing] Which brings up the obvious rival. Can't you just train verbal CoT to be short?

58
00:11:45,725 --> 00:12:13,961
[Dr. Ada Shannon] That's the missing control. L1 uses RL with a length target to make a reasoning model hit a token budget, and budget forcing and length-penalty RL do similar things. None are run here. The 32-token truncation test cuts a verbal trace off after the fact, which isn't a verbal policy trained to be short. The experiment that settles it is verbal GRPO with a length penalty at about 144 tokens. If it matches, the vocabulary adds nothing beyond bounding length.

59
00:12:13,961 --> 00:12:17,026
[Hal Turing] The continuous methods are missing too, right? Coconut, CODI?

60
00:12:17,026 --> 00:12:34,441
[Dr. Ada Shannon] Right. CODI is the closest relative, since its self-distillation resembles this warm-up. Neither it nor Coconut is a baseline. So the claim that discrete tokens match continuous latents is untested here. Only pause tokens and Stepwise Internalization are run, and neither gets an RL stage.

61
00:12:34,441 --> 00:12:37,599
[Hal Turing] So the win over pause tokens might be the RL, not the tokens.

62
00:12:37,599 --> 00:13:10,385
[Dr. Ada Shannon] Possibly. Warm-up alone gets 88.0, and RL lifts it to 90.8. The cleanest control is the same RL on a direct-answer model with no abstract tokens. The pause baseline is also fine-tune-only, which Goyal's work says is the weak setting. Separately, Pfau, Merrill and Bowman's Let's Think Dot by Dot, from NYU in 2024, found filler positions help mainly on parallelizable tasks. So extra compute alone could explain part of this.

63
00:13:10,385 --> 00:13:12,428
[Hal Turing] What would separate compute from content?

64
00:13:12,428 --> 00:13:44,797
[Dr. Ada Shannon] Swap in random tokens, constant tokens, or a trace from a different prompt at test time. The paper's permutation test is weaker. Scrambling the abstract tokens costs about 8 points, versus 11 for scrambled verbal reasoning, so order seems to matter. But the scrambled score, 82.8, lands below the no-CoT model's 85.8. So the trace is either order-sensitive, or a corrupted one misleads the answer. The paper can't say which. And a dominant token like TOKEN_F could be padding-like, with no meaning at all.

65
00:13:44,797 --> 00:13:47,723
[Hal Turing] The paper also pitches monitorability. Does that survive?

66
00:13:47,723 --> 00:14:07,878
[Dr. Ada Shannon] I'd argue the opposite. Korbak and colleagues, in 2025, argued that readable chains of thought are a fragile safety opportunity. Replacing words that may be unfaithful with unreadable tokens doesn't fix faithfulness. There's no probing or token-swap experiment here, so monitorability is a hope.

67
00:14:07,878 --> 00:14:09,132
[Hal Turing] Then there's AIME.

68
00:14:09,132 --> 00:14:38,992
[Dr. Ada Shannon] It's the clearest boundary. On AIME'25, the total is 3,438 tokens, and with the trace capped at 128, about 3,300 must be response. So the response is doing the long reasoning, and compression there is 2.7x, not 12x. On GPQA it's 7.9x, with accuracy about one point lower. On AIME the gap is 24.4 versus 25.6. With single runs and no intervals reported, gaps that small could be noise.

69
00:14:38,992 --> 00:14:40,618
[Hal Turing] What else does the paper leave out?

70
00:14:40,618 --> 00:14:56,222
[Dr. Ada Shannon] Efficiency is only measured in token counts. There's no wall-clock latency. The reward comes from a generative model, gpt-oss-20b, and a 600k-sample post-training run plus a million RL episodes needs a payback argument the paper doesn't make.

71
00:14:56,222 --> 00:14:58,219
[Hal Turing] Where does that leave practitioners?

72
00:14:58,219 --> 00:15:26,268
[Dr. Ada Shannon] If you serve short-horizon reasoning at volume, it's worth a pilot, but measure latency yourself and run the length-penalized verbal baseline first. For future work, I'd want budget-adaptive trace length, probing of what each token encodes, and the random-trace control. Gist tokens, from Jesse Mu and colleagues at Stanford in 2023, already showed a small set of learned tokens can act as a bottleneck. The new part here is doing it for reasoning, with RL.

73
00:15:26,268 --> 00:15:47,491
[Hal Turing] So here's where I land. This establishes a post-training-only recipe where a discrete bottleneck keeps most of verbal CoT's accuracy at a small token count on MATH-500. What it doesn't show is that the gain comes from the abstract vocabulary rather than from bounding length, or that discrete beats continuous latents.

74
00:15:47,491 --> 00:15:49,349
[Dr. Ada Shannon] That's accurate. Thanks for listening.

75
00:15:49,349 --> 00:15:51,253
[Hal Turing] Thanks, everyone. See you next time.