One frozen backbone runs twice. The gap between the two MLP outputs is written into fast weights. Step through it.
Lineage of what gets written from episode
Hover a node. Colors mark the training objective for the state.
Hover a method.
Teacher vs student window Illustrative
Causal attention grid (rows = query, columns = key) at a toy scale of 20 tokens. Orange cells are the residual: visible to the teacher, not the student. Dark cells are in neither pass.
Hover a cell.
Paper settings (from scratch, 340M/760M): teacher 2048, student 1024. Retrofit: teacher full attention on 64K, student 2048. No teacher/student ratio sweep is reported.
The write: residual × activation Illustrative values
Eq. 4 pulls student MLP output toward the detached teacher output. Its gradient wrt the down-projection gives an outer product (Eq. 6): residual is the value, student activation is the address. Cell values here are made up to show structure.
Hover a cell of the update matrix.
Loss (Eq. 4) vs what runs (Eq. 12) from episode
Reported results numbers from episode
Single runs, no intervals. Values marked ≈ are approximate readings quoted in the episode.