Self-Improving Pretraining With Post-Trained Models

Upstream Preference Injection

The core move is not “post-train harder.” It is to treat continuation quality as a sequence-level object during pretraining and let a stronger post-trained model reshape what the learner sees early.

3-way

each prefix compares corpus suffix, teacher rewrite, and learner rollouts

K-token

continuations become judged chunks, not isolated next-token guesses

2 roles

the strong model acts as generator and judge surrogate in the loop

original corpus material teacher rewrite and judge signal learner self-rollouts and RL update

Sequence Tournament

Instead of one token target, the learner faces a judged continuation contest. Use the phase selector to see how dependence shifts from teacher-heavy bootstrapping toward learner-produced rollouts.

Early

teacher rewrite dominates quality and safety margins

Mid

learner rollouts begin to win on local coherence

Late

RL mostly sharpens the learner’s own judged completions

low judge score medium judged value high preference pressure

Claimed Improvement Profile

The transcript reports separate gains for continual pretraining and from-scratch runs. Toggle views to see how the paper’s headline changes depending on whether you look at relative gains or the absolute quality jump.

36.2%

relative factuality improvement in continual pretraining

86.3%

quality win-rate gain ceiling mentioned in the episode

31.1 pts

absolute quality win-rate gain in from-scratch runs

Idea Lineage And Pressure Points

This method looks novel in assembly more than ingredients. The diagram maps what it inherits from earlier work, where it pushes those ideas earlier, and where the skepticism clusters around judges, reward hacking, and teacher style transfer.

CAI

AI feedback and revision logic

STaR

self-generated reasoning used as training fuel

RLHF risk

judge optimization can drift into confidence theater