Stochastic Weight Averaging: Wider Optima and Better Generalization

Izmailov, Podoprikhin, Garipov, Vetrov, Wilson · 2018 (UAI) · Visual companion to the podcast episode
arXiv 1803.05407 Interactive page

The SWA recipe

Click a step. Four moves: train, wander, average, repair BatchNorm.

Learning-rate schedules

Baseline schedule as described in the episode vs the SWA continuation. SWA rate levels: Illustrative

Serving cost: ensemble vs one averaged model

Test-time cost of an ensemble scales linearly with members (FGE typically collects six to twelve).

Why average? Iterates sit on a rim

Schematic loss bowl with noisy high-rate iterates on a shell. Illustrative, synthetic points

Train vs test surfaces are shifted

A narrow minimum is fragile under the shift; a wide one stays good. Illustrative toy curves

Reported results

Numbers quoted in the episode. Bar axes may not start at zero; the axis floor is printed.

What does “wide” mean?

Distance walked before loss rises by a threshold. Illustrative curves

The ReLU rescaling caveat (Dinh et al.)

Scale one layer by α and the next by 1/α: same function, different weight-space width. Toy: L=(w1·w2−1)²

Three claims, one question

Click a claim. Tags reflect the episode’s qualitative assessment, not a numeric score.

Is the baseline fair?

Budget layout from the episode. SGD baseline: constant to 0.5, linear decay to 1% by 0.9, then flat.

Controls the paper skipped

References