Finding the Heads That Trigger Non-Linear Reasoning
Five stages, one per-head logistic probe, then a shared-subspace PCA that denoises the raw signal by pooling every head in a layer into one covariance matrix.
Pipeline Flow
Per-Head Probe Accuracy (illustrative grid — 8 layers × 12 heads)
Hover a cell to inspect a head's probe accuracy.
low probe accuracy
mid
high
top-10% candidate (outlined)
Pause, Rotate, Fork: The Live Walkthrough
Converting the point (0, 3) to polar coordinates. Same correct answer, wildly different path length, depending on one rotation applied to one hidden state.
Reasoning Trace
Norm-Preserving Rotation (Eq. 5)
original hidden state (dashed)
rotated → suppress
rotated → amplify
Benchmark Deltas: CREST vs. Vanilla
Bars diverge from zero. Green means CREST wins; orange-flagged bars are the two rows the hosts caught barely moving or moving the wrong way.
← model gets worse · model improves →
Calibrate on Math, Deploy Everywhere
Every probe and steering vector is fit once on MATH500 — 500 problems — then reused with zero recalibration on three domains that share nothing with math but tokens.
Transfer Flow
Headline Transfer Win — GPQA-D, R1-32B
Where Transfer Cracks — Qwen3-30B
Which Ratio Actually Produced the Headline Numbers?
Three sections, three different fractions of heads steered — and the paper never states which one generated Tables 1 through 3.