First-order methods only ever see the gradient. Second-order methods bring in curvature — but the exact form is too expensive to use at scale. Shampoo's whole story is finding a tractable approximation to that curvature.
For N parameters, a full preconditioner (Hessian or full-matrix AdaGrad) costs O(N²) to store and O(N³) to invert. Shampoo's Kronecker factoring collapses this back toward the per-dimension sizes.
A diagonal preconditioner (Adam) can only stretch each coordinate independently. A full-matrix preconditioner can also rotate — capturing correlation between parameters. Hover any cell to inspect it.
Instead of one N×N monster, Shampoo keeps one small matrix per tensor dimension — L for rows, R for columns — and approximates the full preconditioner as their Kronecker product L ⊗ R.
Shampoo isn't alone in this game. K-FAC factors the Fisher information matrix; K-BFGS factors the Hessian directly. All three only get head-to-head tested at toy scale in this paper.
Real convergence gains across four production-scale workloads — but wall-clock savings only materialize where the systems engineering caught up, as BERT-Large shows.
The only head-to-head second-order comparison in the paper — MNIST / FACES / CURVES autoencoders, tens of thousands of parameters. All three factored methods cluster together, well ahead of RMSprop/Adam.
Three extra chunks of work versus Adam. The inverse fourth-root computation is the killer — up to 100× the cost of a normal step, and it must run in double precision because L and R are badly ill-conditioned.
The loss landscape doesn't shift much step to step, so the inverse root is only recomputed every few hundred steps — and it's offloaded to idle CPUs running alongside the accelerator, computed asynchronously and swapped in when ready.