Abstract Chain-of-Thought: Reasoning with Discrete Latent Tokens

Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
Keshav Ramji, Tahira Naseem, Ramón Fernandez Astudillo (IBM Research AI) · 2026
arXiv:2604.22709 Interactive page

Verbal vs. abstract reasoning

Verbal CoT writes words. Abstract-CoT writes up to 128 tokens from a new 64-token codebook, then answers.

The recipe, stage by stage

Click a stage. Post-training only: no pre-training run, no architecture change.

Where it sits among predecessors

Schematic positioning, not measured data. Illustrative

The attention-mask bottleneck Illustrative layout

Rows = query tokens, columns = keys they may read. Token counts are schematic; the mask rule follows the paper's description.

Hover a cell or row.

Warm-up loop: who writes the abstract tokens?

Three iterations. Step through them.

Constrained decoding of the abstract segment Illustrative

Drag to choose the position t in the abstract segment. Allowed set = 64 codebook tokens + end delimiter, renormalized; at 128 the delimiter is forced.

MATH-500, Qwen3-8B

Reported values. Verbal SFT+RL numbers are approximate (~93 accuracy, ~1,671 tokens). Token column = reasoning + response.

Token compression across benchmarks

Ratio of verbal-baseline tokens to Abstract-CoT tokens as stated in the episode. Hatched bars are lower bounds ("a bit over four", "under half").

Hover a bar.

Accuracy gaps are small

Single runs, no variance reported. Treat small gaps as unresolved.

Ablation ladder on MATH-500 (Qwen3-8B)

Accuracy axis starts at 70 to show the differences. Pick a question to highlight the relevant bars.

Scrambling test

Reported drops: about 8 points for abstract tokens (to 82.8), about 11 for scrambled verbal reasoning.

82.8 sits below the 85.8 no-CoT model: order-sensitivity and a misleading corrupted trace cannot be separated here.

What is the 144?

Table 1 counts reasoning + response tokens together. The cap of 128 applies only to the abstract segment.

Token usage: uniform to skewed Illustrative

The paper reports uniform initial usage and Zipf-like skew after RL, with TOKEN_F far ahead. These counts are a schematic, not paper data.

Reuse is not meaning: a dominant token might be a useful habit, or a padding-like default.

Controls the episode asks for

Which comparisons does the paper actually run?

Hover a row.

References

  1. Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought — Ramji, Naseem, Astudillo (arXiv 2604.22709)
  2. Coconut: Training LLMs to Reason in a Continuous Latent Space — Hao et al.
  3. CODI: Compressing CoT into Continuous Space via Self-Distillation — Shen et al.
  4. Think before you speak: Pause Tokens — Goyal et al.
  5. L1: Controlling How Long a Reasoning Model Thinks — Aggarwal, Welleck
  6. Let's Think Dot by Dot — Pfau, Merrill, Bowman
  7. Chain of Thought Monitorability — Korbak et al.
  8. Soft Tokens, Hard Truths — Butt et al.
  9. Learning to Compress Prompts with Gist Tokens — Mu et al.
  10. Language Models Don't Always Say What They Think — Turpin et al.