Why Open Relational Foundation Models Fail
Jun 24, 2026
This episode explores why open relational foundation models struggle on real downstream database tasks, using OpenRFM as a case study in relational in-context learning across multi-table data such as healthcare, fraud, and recommendation systems. It explains how the RT backbone builds breadth-first relational contexts, why that setup often reduces to a kernel-regression-like similarity lookup, and how limited labeled evidence in those walks creates a label-scarcity bottleneck. The discussion highlights the paper’s main argument that both architecture and pretraining prior matter: adding a TabICL head gives the query direct access to a full batch of support examples, while better synthetic and real-data pretraining pushes the model from shallow similarity matching toward actual relational feature learning. Listeners would find it interesting because the episode goes beyond benchmark gains to unpack a concrete failure mode, then shows how OpenRFM turns that diagnosis into a reported roughly 30% average improvement over the RT baseline.
Sources:
1. OpenRFM: Dissecting Relational In-Context Learning — Zhikai Chen, Junyu Yin, Jialiang Gu, Siheng Xiong, Xiaoze Liu, Ruowang Zhang, Keren Zhou, Kai Guo, 2026
http://arxiv.org/abs/2606.04320
2. Neural Tangent Kernel: Convergence and Generalization in Neural Networks — Arthur Jacot, Franck Gabriel, Clement Hongler, 2018
https://arxiv.org/abs/1806.07572
3. On Lazy Training in Differentiable Programming — Lenaic Chizat, Edouard Oyallon, Francis Bach, 2019
https://arxiv.org/abs/1812.07956
4. What learning algorithm is in-context learning? Investigations with linear models — Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou, 2022
https://arxiv.org/abs/2211.15661
5. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second — Noah Hollmann, Samuel Muller, Katharina Eggensperger, Frank Hutter, 2022
https://arxiv.org/abs/2207.01848
6. Assortative mixing in networks — M. E. J. Newman, 2002
https://arxiv.org/abs/cond-mat/0205405
7. Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs — Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, Danai Koutra, 2020
https://arxiv.org/abs/2006.11468
8. Graph Neural Networks with Heterophily — Jiong Zhu, Ryan A. Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K. Ahmed, Danai Koutra, 2020
https://arxiv.org/abs/2009.13566
9. PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models — Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik, Vijay Prakash Dwivedi, Johannes Hoffart, Carlos Guestrin, Jure Leskovec, 2026
https://arxiv.org/abs/2602.04029
10. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data — Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos I. Kanatsoulis, Roshan Reddy Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, and Jure Leskovec, 2026
https://scholar.google.com/scholar?q=Relational+Transformer%3A+Toward+Zero-Shot+Foundation+Models+for+Relational+Data
11. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data — Jingang Qu, David Holzmuller, Gael Varoquaux, and Marine Le Morvan, 2025
https://scholar.google.com/scholar?q=TabICL%3A+A+Tabular+Foundation+Model+for+In-Context+Learning+on+Large+Data
12. KumoRFM: A Foundation Model for In-Context Learning on Relational Data — Matthias Fey, Vid Kocijan, Federico Lopez, Jan Eric Lenssen, and Jure Leskovec, 2025
https://scholar.google.com/scholar?q=KumoRFM%3A+A+Foundation+Model+for+In-Context+Learning+on+Relational+Data
13. RelBench: A Benchmark for Deep Learning on Relational Databases — Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec, 2024
https://scholar.google.com/scholar?q=RelBench%3A+A+Benchmark+for+Deep+Learning+on+Relational+Databases
14. Understanding Emergent In-Context Learning from a Kernel Regression Perspective — Chi Han, Ziqi Wang, Han Zhao, and Heng Ji, 2025
https://scholar.google.com/scholar?q=Understanding+Emergent+In-Context+Learning+from+a+Kernel+Regression+Perspective
15. Retrieval & Fine-Tuning for In-Context Tabular Models — Valentin Thomas et al., 2024
https://scholar.google.com/scholar?q=Retrieval+%26+Fine-Tuning+for+In-Context+Tabular+Models
16. On Finetuning Tabular Foundation Models — Ivan Rubachev et al., 2025
https://scholar.google.com/scholar?q=On+Finetuning+Tabular+Foundation+Models
17. Turning Tabular Foundation Models into Graph Foundation Models — Dmitry Eremeev et al., 2025
https://scholar.google.com/scholar?q=Turning+Tabular+Foundation+Models+into+Graph+Foundation+Models
18. Of Graphs and Tables: Zero-Shot Node Classification with Tabular Foundation Models — Adrian Hayler et al., 2025
https://scholar.google.com/scholar?q=Of+Graphs+and+Tables%3A+Zero-Shot+Node+Classification+with+Tabular+Foundation+Models
19. A Pre-training Framework for Relational Data with Information-theoretic Principles — Quang Truong et al., 2025
https://scholar.google.com/scholar?q=A+Pre-training+Framework+for+Relational+Data+with+Information-theoretic+Principles
20. When Heterophily Meets Heterogeneity: New Graph Benchmarks and Effective Methods — Junhong Lin et al., 2024
https://scholar.google.com/scholar?q=When+Heterophily+Meets+Heterogeneity%3A+New+Graph+Benchmarks+and+Effective+Methods
21. Aligning the Spectrum: Hybrid Graph Pre-training and Prompt Tuning across Homophily and Heterophily — Haitong Luo et al., 2025
https://scholar.google.com/scholar?q=Aligning+the+Spectrum%3A+Hybrid+Graph+Pre-training+and+Prompt+Tuning+across+Homophily+and+Heterophily
22. AI Post Transformers: Can Models Learn from Long Context? — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-05-can-models-learn-from-long-context-77533e.mp3
23. AI Post Transformers: When Many-Shot CoT Becomes Test-Time Learning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-15-when-many-shot-cot-becomes-test-time-lea-c25bfe.mp3
24. AI Post Transformers: TransactionGPT as a Payments Foundation Model — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-17-transactiongpt-as-a-payments-foundation-d53687.mp3
1. OpenRFM: Dissecting Relational In-Context Learning — Zhikai Chen, Junyu Yin, Jialiang Gu, Siheng Xiong, Xiaoze Liu, Ruowang Zhang, Keren Zhou, Kai Guo, 2026
http://arxiv.org/abs/2606.04320
2. Neural Tangent Kernel: Convergence and Generalization in Neural Networks — Arthur Jacot, Franck Gabriel, Clement Hongler, 2018
https://arxiv.org/abs/1806.07572
3. On Lazy Training in Differentiable Programming — Lenaic Chizat, Edouard Oyallon, Francis Bach, 2019
https://arxiv.org/abs/1812.07956
4. What learning algorithm is in-context learning? Investigations with linear models — Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou, 2022
https://arxiv.org/abs/2211.15661
5. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second — Noah Hollmann, Samuel Muller, Katharina Eggensperger, Frank Hutter, 2022
https://arxiv.org/abs/2207.01848
6. Assortative mixing in networks — M. E. J. Newman, 2002
https://arxiv.org/abs/cond-mat/0205405
7. Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs — Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, Danai Koutra, 2020
https://arxiv.org/abs/2006.11468
8. Graph Neural Networks with Heterophily — Jiong Zhu, Ryan A. Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K. Ahmed, Danai Koutra, 2020
https://arxiv.org/abs/2009.13566
9. PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models — Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik, Vijay Prakash Dwivedi, Johannes Hoffart, Carlos Guestrin, Jure Leskovec, 2026
https://arxiv.org/abs/2602.04029
10. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data — Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos I. Kanatsoulis, Roshan Reddy Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, and Jure Leskovec, 2026
https://scholar.google.com/scholar?q=Relational+Transformer%3A+Toward+Zero-Shot+Foundation+Models+for+Relational+Data
11. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data — Jingang Qu, David Holzmuller, Gael Varoquaux, and Marine Le Morvan, 2025
https://scholar.google.com/scholar?q=TabICL%3A+A+Tabular+Foundation+Model+for+In-Context+Learning+on+Large+Data
12. KumoRFM: A Foundation Model for In-Context Learning on Relational Data — Matthias Fey, Vid Kocijan, Federico Lopez, Jan Eric Lenssen, and Jure Leskovec, 2025
https://scholar.google.com/scholar?q=KumoRFM%3A+A+Foundation+Model+for+In-Context+Learning+on+Relational+Data
13. RelBench: A Benchmark for Deep Learning on Relational Databases — Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec, 2024
https://scholar.google.com/scholar?q=RelBench%3A+A+Benchmark+for+Deep+Learning+on+Relational+Databases
14. Understanding Emergent In-Context Learning from a Kernel Regression Perspective — Chi Han, Ziqi Wang, Han Zhao, and Heng Ji, 2025
https://scholar.google.com/scholar?q=Understanding+Emergent+In-Context+Learning+from+a+Kernel+Regression+Perspective
15. Retrieval & Fine-Tuning for In-Context Tabular Models — Valentin Thomas et al., 2024
https://scholar.google.com/scholar?q=Retrieval+%26+Fine-Tuning+for+In-Context+Tabular+Models
16. On Finetuning Tabular Foundation Models — Ivan Rubachev et al., 2025
https://scholar.google.com/scholar?q=On+Finetuning+Tabular+Foundation+Models
17. Turning Tabular Foundation Models into Graph Foundation Models — Dmitry Eremeev et al., 2025
https://scholar.google.com/scholar?q=Turning+Tabular+Foundation+Models+into+Graph+Foundation+Models
18. Of Graphs and Tables: Zero-Shot Node Classification with Tabular Foundation Models — Adrian Hayler et al., 2025
https://scholar.google.com/scholar?q=Of+Graphs+and+Tables%3A+Zero-Shot+Node+Classification+with+Tabular+Foundation+Models
19. A Pre-training Framework for Relational Data with Information-theoretic Principles — Quang Truong et al., 2025
https://scholar.google.com/scholar?q=A+Pre-training+Framework+for+Relational+Data+with+Information-theoretic+Principles
20. When Heterophily Meets Heterogeneity: New Graph Benchmarks and Effective Methods — Junhong Lin et al., 2024
https://scholar.google.com/scholar?q=When+Heterophily+Meets+Heterogeneity%3A+New+Graph+Benchmarks+and+Effective+Methods
21. Aligning the Spectrum: Hybrid Graph Pre-training and Prompt Tuning across Homophily and Heterophily — Haitong Luo et al., 2025
https://scholar.google.com/scholar?q=Aligning+the+Spectrum%3A+Hybrid+Graph+Pre-training+and+Prompt+Tuning+across+Homophily+and+Heterophily
22. AI Post Transformers: Can Models Learn from Long Context? — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-05-can-models-learn-from-long-context-77533e.mp3
23. AI Post Transformers: When Many-Shot CoT Becomes Test-Time Learning — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-05-15-when-many-shot-cot-becomes-test-time-lea-c25bfe.mp3
24. AI Post Transformers: TransactionGPT as a Payments Foundation Model — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-06-17-transactiongpt-as-a-payments-foundation-d53687.mp3