On-Policy Self-Distillation Using Majority Vote Instead of Ground Truth

Li, Wang, Liang, Tian, Fu, Vasconcelos · 2026 preprint (U-OPSD) · Qwen3 4B/8B on competition math
arXiv 2608.06296 Interactive page

The ladder of training signals

Pick a method. Token cells show how feedback reaches each token of a rollout. Illustrative credit pattern

Where U-OPSD sits

Same-network teacher, but the context comes from the model's own vote.

Step-by-step: one prompt becomes a training signal

Vote simulator Illustrative rollouts

Tap a rollout to cycle its answer: A → B → C → truncated (no answer). The score is the winner's share of all 8.
0.50

Average accuracy: label-free comparison

Does it win everywhere?

Reported exceptions and edges, in accuracy points.

Ablations (8B non-thinking, five-benchmark average)

One configuration, best checkpoint. Read as a map of mechanism, not precise effect sizes.

One problem = 3.3 points

AIME has 30 problems. Tap cells to mark solved.

Best-of-6 checkpoints bias Illustrative

Simulated: equal true skill, noisy evals. Max over 6 checkpoints beats a fixed final step.

Is the OPSD comparison budget-matched?

Per-step settings from the setup text.

Pseudo-label quality probe

64-prompt probe: pseudo-labels matched gold 86.7%. It never trains on deliberately corrupted votes.