From ProphetNet's broken joints through DeepSeek's partial chains to Draxler et al.'s full joint prediction via normalizing flows
In standard autoregressive decoding, the model produces a distribution π over the vocabulary, then a uniform random variable u ~ U[0,1] is drawn after the forward pass and fed into the inverse CDF ("Pick" function) to select a token. That u just floats there, sampled outside the model's reasoning.
When ProphetNet, Medusa, or Gloeckle/Meta MTP try to predict multiple tokens at once, each head independently predicts its token given only the shared trunk. They model P(t2|z) and P(t3|z) separately — but the true target is P(t2,t3|context) , a joint.
"Name a capital of South Africa" — Cape Town and Pretoria are both valid. An independent MTP model can pick "C" from Cape Town at Head1 and "retoria" from Pretoria at Head2. Result: Cretoria — token-salad with near-zero joint probability.
Independent heads on a shared encoder. Each future position predicted without knowledge of the others. First major MTP attempt — fully breaks the joint distribution.
Tree-structured speculative heads for self-speculative decoding. Each head independently predicts a future token — same independence flaw, arranged in a tree for verification.
n independent output heads on a shared transformer trunk. Proven gains as training signal (12–17% on HumanEval/MBPP) — the extra heads force richer intermediate representations. But at inference, each head is blind to the others' choices. The extra heads are valuable as training signal, not for joint prediction.
Instead of independent heads, DeepSeek chains them: the k-th MTP module takes the (k−1)-th module's representation concatenated with the next token's embedding via linear projection. Each depth conditions on the previous. Partial — each module is a single lightweight layer, capturing shallow conditioning.
Uses the DeepSeek-style sequential causal chain, NOT Gloeckle's independent heads. The Megatron-LM docs describe the k-th MTP module combining the representation from depth k−1 with the embedding of the next token via linear projection. Same partial dependency structure.
Resolves the independence flaw entirely. By feeding auxiliary random variables u ~ U[0,1] into the model alongside token embeddings, the model knows which token will be sampled at each position. Theorem 2 proves this captures the full autoregressive joint. No information is lost.
The fundamental insight: instead of sampling u after the forward pass, feed it into the forward pass alongside the token embedding. The model no longer predicts "what token is likely next" — it predicts "given this specific random draw, what token gets selected, and what does that imply for everything downstream."
O-PTP (Theorem 1): Given context and ALL auxiliary variables ui...uk, the model deterministically outputs tk = argmax P_theta(.|context, ui,...,uk). Each token is fully determined by its u — the model learns a one-hot mapping. Fast and simple, but doesn't expose the full probability distribution.
C-PTP (Theorem 2) — The Key Result: By conditioning the prediction of token tk on context and all preceding auxiliary variables ui,...,u(k-1) but excluding uk, we get P(tk|context,ui,...,u(k-1)) = P(tk|t<k). Conditioning on previous u's is equivalent to conditioning on the actual sampled tokens. This recovers the full conditional distribution while maintaining parallel computation. This is the joint prediction that Theorem 2 addresses cleanly — and that all prior methods broke.
The transformer is re-trained to predict something fundamentally different: not "what token is likely next" but "given a specific random draw, what token gets selected and what does that imply downstream."
| Approach | Year | Dependencies | Architecture Style |
|---|---|---|---|
| ProphetNet | 2020 | None | Independent n-gram heads |
| Medusa | 2024 | None | Independent tree-structured heads |
| Gloeckle / Meta FAIR | 2024 | None | Independent output heads (n=4) |
| DeepSeek V3 | 2024 | Partial | Sequential causal chain across depths |
| Nemotron 3 (NVIDIA) | 2025 | Partial | Sequential causal chain (DeepSeek-style) |
| PTP (Draxler et al.) | 2025 | Full | Joint dist. via normalizing flows + aux vars |
Feeding u auxiliaries alongside token embeddings changes what the transformer processes per call. If those inputs extend the effective sequence or require additional attention positions, cache reuse patterns change. A model that reduces sequential steps but significantly increases per-step KV cache footprint may not yield practical latency gains under memory-constrained serving. The paper does not analyze this tradeoff.
One single model tested (distilled from Vicuna-7B / TinyLlama-1.1B). No throughput benchmarks. The accepted deployment path is speculative decoding — a verifier is still required — so the speedup is bounded by verifier cost. Much more testing needed across model scales.
The extra u inputs increase the effective sequence length the model must process. In memory-bandwidth-bound regimes (small batch), latency wins might hold. In compute-bound regimes (large batch), throughput impact is unclear and untested.
Theorem 2 cleanly addresses the joint prediction problem — conditioning on previous u's is provably equivalent to conditioning on previous tokens. First framework to achieve full joint dependency preservation in parallel token generation. The normalizing-flow bridge from IAF is elegant and the proofs are clean.