Weak-SIGReg for Stable Vision Transformer Training

This episode explores Weak-SIGReg, a lightweight covariance regularizer designed to prevent representation collapse in fragile supervised training, especially for small-data Vision Transformers. It explains how the method uses a sketched covariance matrix and an identity-matching penalty to keep hidden features decorrelated and similarly scaled at much lower cost than full covariance regularization. The discussion centers on CIFAR-100 results, where Weak-SIGReg dramatically improves a deliberately unstable ViT setup and also boosts a plain MLP, while offering little change on an already stable ResNet18. It also digs into the paper’s main caveat: the biggest gains appear when the baseline training recipe is badly broken, so the most interesting question is not just whether the regularizer works, but when it adds real value beyond simply fixing optimization and initialization.

Sources:
1. Weak-SIGReg: Covariance Regularization for Stable Deep Learning — Habibullah Akbar, 2026
http://arxiv.org/abs/2603.05924
2. Reducing Overfitting in Deep Networks by Decorrelating Representations — Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, Dhruv Batra, 2015
https://scholar.google.com/scholar?q=Reducing+Overfitting+in+Deep+Networks+by+Decorrelating+Representations
3. Barlow Twins: Self-Supervised Learning via Redundancy Reduction — Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stephane Deny, 2021
https://scholar.google.com/scholar?q=Barlow+Twins%3A+Self-Supervised+Learning+via+Redundancy+Reduction
4. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021
https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning
5. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — Randall Balestriero, Yann LeCun, 2025
https://scholar.google.com/scholar?q=LeJEPA%3A+Provable+and+Scalable+Self-Supervised+Learning+Without+the+Heuristics
6. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al., 2021
https://scholar.google.com/scholar?q=An+Image+is+Worth+16x16+Words%3A+Transformers+for+Image+Recognition+at+Scale
7. Training data-efficient image transformers & distillation through attention — Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herve Jegou, 2021
https://scholar.google.com/scholar?q=Training+data-efficient+image+transformers+%26+distillation+through+attention
8. How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers — Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer, 2021
https://scholar.google.com/scholar?q=How+to+train+your+ViT%3F+Data%2C+Augmentation%2C+and+Regularization+in+Vision+Transformers
9. Early Convolutions Help Transformers See Better — Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollar, Ross Girshick, 2021
https://scholar.google.com/scholar?q=Early+Convolutions+Help+Transformers+See+Better
10. Whitening for Self-Supervised Representation Learning — Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe, 2020
https://arxiv.org/abs/2007.06346
11. Understanding Dimensional Collapse in Contrastive Self-Supervised Learning — Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian, 2021
https://arxiv.org/abs/2110.09348
12. UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures — Triet M. Le, 2026
https://arxiv.org/abs/2606.01443
13. Stability of Transformers under Layer Normalization — Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Krishna Kumar, Markos A. Katsoulakis, 2025
https://scholar.google.com/scholar?q=Stability+of+Transformers+under+Layer+Normalization
14. Transformers without Normalization — Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, Zhuang Liu, 2025
https://scholar.google.com/scholar?q=Transformers+without+Normalization
15. Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations — Yongyi Yang, Jacob Steinhardt, Wei Hu, 2023
https://scholar.google.com/scholar?q=Are+Neurons+Actually+Collapsed%3F+On+the+Fine-Grained+Structure+in+Neural+Representations
16. The Impact of Geometric Complexity on Neural Collapse in Transfer Learning — Michael Munn, Benoit Dherin, Javier Gonzalvo, 2024
https://scholar.google.com/scholar?q=The+Impact+of+Geometric+Complexity+on+Neural+Collapse+in+Transfer+Learning
17. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3
18. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp3