Toggle the recipe. VPO only changes the advantage computation and what counts as a sample.
Search needs a varied pool
Best-of-N and evolutionary search can only exploit differences between candidates.Illustrative
Reward vectors and the Pareto frontier
Each dot is a candidate scored on two components (e.g. gold vs diamond). A scalarization weight picks one dot.Illustrative points
Scoring a set of three answers
Set score = average over K shared weight draws of the best answer's w·r. Step through it.Illustrative numbers
Why both pieces are needed
Paper's ablation logic as described on the episode, drawn as a 2×2.
Reported results
Numbers quoted from the episode. One training run per table cell, no confidence intervals.Reported
Reward-space diversity
Mean pairwise L1 distance between candidates' reward vectors. Reworded copies score ~0.Toy vectors
When VPO has nothing to diversify
As reward components become collinear, the Pareto front shrinks toward one point. Paper: on-policy correlation 0.95 on UltraFeedback, VPO 0.776 vs GRPO-style 0.779 best@50.Synthetic cloud