Squisher: Approximating the Fisher Information Matrix and use cases

We focus on the July 2025 paper, "Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator". The paper goes into the mathematical details of approximating the FIM, and the easiest is what they call the Squisher: the Adam optimizer variance. This technique allows for tasks like model pruning and model merging to be performed "for free" without the significant computational overhead typically required to calculate the Fisher Information Matrix. We also review the old 1992 paper "Second order derivatives for network pruning: Optimal Brain Surgeon" in terms of what was missing in light of the Squisher paper.