OOD Shifts Make LLM Representations Sparser
Apr 2, 2026
This episode explores a March 19, 2026 study on whether large language models respond to out-of-distribution prompts by compressing their internal activity into fewer active dimensions. It explains how the paper connects two traditions in AI research, mechanistic interpretability and representation geometry, by proposing hidden-state sparsity as a measurable internal signature of stress from harder reasoning tasks, longer contexts, and conflicting information. The discussion breaks down the paper’s core metrics, including Top-k Energy and L1 norm, and clarifies why sparser activations should not be treated as proof of better reasoning or cleaner representations. Listeners would find it interesting because it ties abstract internal model behavior to practical questions about robustness, reliability, and how to evaluate language models beyond just whether their final answers look correct.
Sources:
1. Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs — Mingyu Jin, Yutong Yin, Jingcheng Niu, Qingcheng Zeng, Wujiang Xu, Mengnan Du, Wei Cheng, Zhaoran Wang, Tianlong Chen, Dimitris N. Metaxas, 2026
http://arxiv.org/abs/2603.03415
2. Domain Generalization: A Survey — Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, Chen Change Loy, 2021
https://scholar.google.com/scholar?q=Domain+Generalization%3A+A+Survey
3. Invariant Risk Minimization — Martin Arjovsky, Leon Bottou, Ishaan Gulrajani, David Lopez-Paz, 2019
https://scholar.google.com/scholar?q=Invariant+Risk+Minimization
4. In Search of Lost Domain Generalization — Ishaan Gulrajani, David Lopez-Paz, 2021
https://scholar.google.com/scholar?q=In+Search+of+Lost+Domain+Generalization
5. WILDS: A Benchmark of in-the-Wild Distribution Shifts — Pang Wei Koh, Shiori Sagawa, Henrik Marklund and many others, 2021
https://scholar.google.com/scholar?q=WILDS%3A+A+Benchmark+of+in-the-Wild+Distribution+Shifts
6. Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images — Bruno A. Olshausen, David J. Field, 1996
https://scholar.google.com/scholar?q=Emergence+of+Simple-Cell+Receptive+Field+Properties+by+Learning+a+Sparse+Code+for+Natural+Images
7. Deep Sparse Rectifier Neural Networks — Xavier Glorot, Antoine Bordes, Yoshua Bengio, 2011
https://scholar.google.com/scholar?q=Deep+Sparse+Rectifier+Neural+Networks
8. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — Armen Aghajanyan, Sonal Gupta, Luke Zettlemoyer, 2021
https://scholar.google.com/scholar?q=Intrinsic+Dimensionality+Explains+the+Effectiveness+of+Language+Model+Fine-Tuning
9. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — Trenton Bricken, Adly Templeton, Joshua Batson and many others, 2023
https://scholar.google.com/scholar?q=Towards+Monosemanticity%3A+Decomposing+Language+Models+With+Dictionary+Learning
10. Understanding Intermediate Layers Using Linear Classifier Probes — Guillaume Alain, Yoshua Bengio, 2017
https://scholar.google.com/scholar?q=Understanding+Intermediate+Layers+Using+Linear+Classifier+Probes
11. Deep Contextualized Word Representations — Matthew E. Peters, Mark Neumann, Mohit Iyyer and others, 2018
https://scholar.google.com/scholar?q=Deep+Contextualized+Word+Representations
12. A Structural Probe for Finding Syntax in Word Representations — John Hewitt, Christopher D. Manning, 2019
https://scholar.google.com/scholar?q=A+Structural+Probe+for+Finding+Syntax+in+Word+Representations
13. How Contextual Are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings — Kawin Ethayarajh, 2019
https://scholar.google.com/scholar?q=How+Contextual+Are+Contextualized+Word+Representations%3F+Comparing+the+Geometry+of+BERT%2C+ELMo%2C+and+GPT-2+Embeddings
14. The Geometry of Innocent Flesh on the Bone: Syntactic Structure in Sentence Embeddings — John Hewitt and Christopher D. Manning, 2019
https://scholar.google.com/scholar?q=The+Geometry+of+Innocent+Flesh+on+the+Bone%3A+Syntactic+Structure+in+Sentence+Embeddings
15. What Factors Affect the Success of In-Context Learning? Investigating the Role of Model Architecture and Task Features — Jason Wei, Yi Tay, Quoc V. Le, Denny Zhou and others, 2022
https://scholar.google.com/scholar?q=What+Factors+Affect+the+Success+of+In-Context+Learning%3F+Investigating+the+Role+of+Model+Architecture+and+Task+Features
16. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker and others, 2024
https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step
17. Do Language Models Generalize to Longer Contexts? — Yixiao Li and collaborators, 2025
https://scholar.google.com/scholar?q=Do+Language+Models+Generalize+to+Longer+Contexts%3F
18. Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Language Models — Nicola De Cao, Wilker Aziz and Ivan Titov, 2022
https://scholar.google.com/scholar?q=Parameter-Efficient+Prompt+Tuning+Makes+Generalized+and+Calibrated+Language+Models
19. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — Jonathan Frankle and Michael Carbin, 2019
https://scholar.google.com/scholar?q=The+Lottery+Ticket+Hypothesis%3A+Finding+Sparse%2C+Trainable+Neural+Networks
20. Adaptive Mixtures of Local Experts — Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan and Geoffrey E. Hinton, 1991
https://scholar.google.com/scholar?q=Adaptive+Mixtures+of+Local+Experts
21. Curriculum Demonstration Selection for In-Context Learning — approx. recent ICL curriculum-learning authors, recent
https://scholar.google.com/scholar?q=Curriculum+Demonstration+Selection+for+In-Context+Learning
22. Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning — approx. recent ICL curriculum-learning authors, recent
https://scholar.google.com/scholar?q=Let%27s+Learn+Step+by+Step%3A+Enhancing+In-Context+Learning+Ability+with+Curriculum+Learning
23. Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers — approx. recent interpretability authors, recent
https://scholar.google.com/scholar?q=Sparse+but+not+Simpler%3A+A+Multi-Level+Interpretability+Analysis+of+Vision+Transformers
24. Weight-Sparse Transformers Have Interpretable Circuits — approx. recent mechanistic interpretability authors, recent
https://scholar.google.com/scholar?q=Weight-Sparse+Transformers+Have+Interpretable+Circuits
25. AI Post Transformers: Chain-of-Thought Reasoning: A Brittle Mirage? — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/chain-of-thought-reasoning-a-brittle-mirage/
26. AI Post Transformers: Advancing Mechanistic Interpretability with Sparse Autoencoders — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/advancing-mechanistic-interpretability-with-sparse-autoencoders/
27. AI Post Transformers: Measuring LLM Reasoning Effort via Deep-Thinking Tokens — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/measuring-llm-reasoning-effort-via-deep-thinking-tokens/
28. AI Post Transformers: CLUE: Hidden-State Clustering for Non-parametric Verification — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/clue-hidden-state-clustering-for-non-parametric-verification/
29. AI Post Transformers: Inverse IFEval: Unlearning LLM Cognitive Inertia — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/inverse-ifeval-unlearning-llm-cognitive-inertia/
30. AI Post Transformers: Hyper-Scaling LLM Inference with KV Cache Compression — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/hyper-scaling-llm-inference-with-kv-cache-compression/
1. Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs — Mingyu Jin, Yutong Yin, Jingcheng Niu, Qingcheng Zeng, Wujiang Xu, Mengnan Du, Wei Cheng, Zhaoran Wang, Tianlong Chen, Dimitris N. Metaxas, 2026
http://arxiv.org/abs/2603.03415
2. Domain Generalization: A Survey — Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, Chen Change Loy, 2021
https://scholar.google.com/scholar?q=Domain+Generalization%3A+A+Survey
3. Invariant Risk Minimization — Martin Arjovsky, Leon Bottou, Ishaan Gulrajani, David Lopez-Paz, 2019
https://scholar.google.com/scholar?q=Invariant+Risk+Minimization
4. In Search of Lost Domain Generalization — Ishaan Gulrajani, David Lopez-Paz, 2021
https://scholar.google.com/scholar?q=In+Search+of+Lost+Domain+Generalization
5. WILDS: A Benchmark of in-the-Wild Distribution Shifts — Pang Wei Koh, Shiori Sagawa, Henrik Marklund and many others, 2021
https://scholar.google.com/scholar?q=WILDS%3A+A+Benchmark+of+in-the-Wild+Distribution+Shifts
6. Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images — Bruno A. Olshausen, David J. Field, 1996
https://scholar.google.com/scholar?q=Emergence+of+Simple-Cell+Receptive+Field+Properties+by+Learning+a+Sparse+Code+for+Natural+Images
7. Deep Sparse Rectifier Neural Networks — Xavier Glorot, Antoine Bordes, Yoshua Bengio, 2011
https://scholar.google.com/scholar?q=Deep+Sparse+Rectifier+Neural+Networks
8. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — Armen Aghajanyan, Sonal Gupta, Luke Zettlemoyer, 2021
https://scholar.google.com/scholar?q=Intrinsic+Dimensionality+Explains+the+Effectiveness+of+Language+Model+Fine-Tuning
9. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — Trenton Bricken, Adly Templeton, Joshua Batson and many others, 2023
https://scholar.google.com/scholar?q=Towards+Monosemanticity%3A+Decomposing+Language+Models+With+Dictionary+Learning
10. Understanding Intermediate Layers Using Linear Classifier Probes — Guillaume Alain, Yoshua Bengio, 2017
https://scholar.google.com/scholar?q=Understanding+Intermediate+Layers+Using+Linear+Classifier+Probes
11. Deep Contextualized Word Representations — Matthew E. Peters, Mark Neumann, Mohit Iyyer and others, 2018
https://scholar.google.com/scholar?q=Deep+Contextualized+Word+Representations
12. A Structural Probe for Finding Syntax in Word Representations — John Hewitt, Christopher D. Manning, 2019
https://scholar.google.com/scholar?q=A+Structural+Probe+for+Finding+Syntax+in+Word+Representations
13. How Contextual Are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings — Kawin Ethayarajh, 2019
https://scholar.google.com/scholar?q=How+Contextual+Are+Contextualized+Word+Representations%3F+Comparing+the+Geometry+of+BERT%2C+ELMo%2C+and+GPT-2+Embeddings
14. The Geometry of Innocent Flesh on the Bone: Syntactic Structure in Sentence Embeddings — John Hewitt and Christopher D. Manning, 2019
https://scholar.google.com/scholar?q=The+Geometry+of+Innocent+Flesh+on+the+Bone%3A+Syntactic+Structure+in+Sentence+Embeddings
15. What Factors Affect the Success of In-Context Learning? Investigating the Role of Model Architecture and Task Features — Jason Wei, Yi Tay, Quoc V. Le, Denny Zhou and others, 2022
https://scholar.google.com/scholar?q=What+Factors+Affect+the+Success+of+In-Context+Learning%3F+Investigating+the+Role+of+Model+Architecture+and+Task+Features
16. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker and others, 2024
https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step
17. Do Language Models Generalize to Longer Contexts? — Yixiao Li and collaborators, 2025
https://scholar.google.com/scholar?q=Do+Language+Models+Generalize+to+Longer+Contexts%3F
18. Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Language Models — Nicola De Cao, Wilker Aziz and Ivan Titov, 2022
https://scholar.google.com/scholar?q=Parameter-Efficient+Prompt+Tuning+Makes+Generalized+and+Calibrated+Language+Models
19. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — Jonathan Frankle and Michael Carbin, 2019
https://scholar.google.com/scholar?q=The+Lottery+Ticket+Hypothesis%3A+Finding+Sparse%2C+Trainable+Neural+Networks
20. Adaptive Mixtures of Local Experts — Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan and Geoffrey E. Hinton, 1991
https://scholar.google.com/scholar?q=Adaptive+Mixtures+of+Local+Experts
21. Curriculum Demonstration Selection for In-Context Learning — approx. recent ICL curriculum-learning authors, recent
https://scholar.google.com/scholar?q=Curriculum+Demonstration+Selection+for+In-Context+Learning
22. Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning — approx. recent ICL curriculum-learning authors, recent
https://scholar.google.com/scholar?q=Let%27s+Learn+Step+by+Step%3A+Enhancing+In-Context+Learning+Ability+with+Curriculum+Learning
23. Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers — approx. recent interpretability authors, recent
https://scholar.google.com/scholar?q=Sparse+but+not+Simpler%3A+A+Multi-Level+Interpretability+Analysis+of+Vision+Transformers
24. Weight-Sparse Transformers Have Interpretable Circuits — approx. recent mechanistic interpretability authors, recent
https://scholar.google.com/scholar?q=Weight-Sparse+Transformers+Have+Interpretable+Circuits
25. AI Post Transformers: Chain-of-Thought Reasoning: A Brittle Mirage? — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/chain-of-thought-reasoning-a-brittle-mirage/
26. AI Post Transformers: Advancing Mechanistic Interpretability with Sparse Autoencoders — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/advancing-mechanistic-interpretability-with-sparse-autoencoders/
27. AI Post Transformers: Measuring LLM Reasoning Effort via Deep-Thinking Tokens — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/measuring-llm-reasoning-effort-via-deep-thinking-tokens/
28. AI Post Transformers: CLUE: Hidden-State Clustering for Non-parametric Verification — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/clue-hidden-state-clustering-for-non-parametric-verification/
29. AI Post Transformers: Inverse IFEval: Unlearning LLM Cognitive Inertia — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/inverse-ifeval-unlearning-llm-cognitive-inertia/
30. AI Post Transformers: Hyper-Scaling LLM Inference with KV Cache Compression — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/hyper-scaling-llm-inference-with-kv-cache-compression/