Coconut: Reasoning in Continuous Latent Space Instead of Words

Hao, Sukhbaatar, Su, Li, Hu, Weston, Tian — FAIR at Meta & UC San Diego — "Training Large Language Models to Reason in a Continuous Latent Space"
First posted Dec 2024 · episode discusses the fourth version
arXiv:2412.06769 Interactive page

1 · The loop: word vs. vector

Vector cells are a decorative Illustrative stand-in for a hidden state, not real values.

2 · Nearby approaches (hover a row)

Hover or tap a method row.

Curriculum: swap written steps for thoughts

At stage k, the first k language steps become k·c thoughts. Step counts drawn here are an Illustrative layout.

Why training is sequential: n thoughts → n+1 passes

Reported training settings

ProsQA search: one path vs. a frontier

Graph layout Illustrative — only the values 0.33 (lempus, step 1) and 0.87 (rorpus, step 2) come from the paper's worked example. Question here: "Is Tom a scrompus or a gorpus?"

Tree of Thoughts vs. Coconut: where does search live?

Schematic Illustrative

Reported results

Hover or tap a bar.

GSM8k scale: Coconut minus no-reasoning (points)

No CoT baseline or variance was reported for either Llama model.

What survives the controls? (hover a cell)

Hover or tap a cell.

Colors are an Illustrative ordinal read of the hosts' discussion, not a measured score.

ProsQA: the latent-vector claim vs. controls

Numbers from the episode: no-latent variants 95.5, 96.6, 98.2; Coconut 97.0; spread under 3 points.

References