APEX: A High-Performance Learned Index on Persistent Memory

This episode examines APEX, a persistent-memory learned index from researchers at CUHK, MIT, Microsoft Research, and Simon Fraser University, presented at VLDB 2022. It traces the collision of two research threads — Intel's Optane persistent memory, which sits on the DDR bus but requires manual cache-line flushing and fencing to guarantee crash durability, and "learned indexes," which replace B-trees with lightweight regression models trained to predict a key's position. The discussion covers why naive approaches fail: running the learned index ALEX directly on persistent memory offers speed but zero crash safety, while wrapping it in standard transactional logging (PMDK) restores consistency at the cost of saturating scarce write bandwidth. It builds toward APEX's core innovation, "probe-and-stash," a technique designed to avoid the record-shifting that makes both alternatives fragile, enabling claimed 15x faster inserts and roughly 42-millisecond crash recovery. Listeners interested in database internals, memory hierarchies, or the practical gap between promising ML systems research and production-ready engineering will find the trade-offs — and the hosts' friendly disagreement over why learned indexes haven't seen wider industry adoption — a compelling entry point into the topic.

Interactive Visualization: APEX: A High-Performance Learned Index on Persistent Memory

Sources:
1. APEX: A High-Performance Learned Index on Persistent Memory — Baotong Lu, Jialin Ding, Eric Lo, Umar Farooq Minhas, Tianzheng Wang, 2021
http://arxiv.org/abs/2105.00683
2. The Case for Learned Index Structures — Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis, 2018
https://scholar.google.com/scholar?q=The+Case+for+Learned+Index+Structures
3. ALEX: An Updatable Adaptive Learned Index — Jialin Ding, Umar Farooq Minhas, Jia Yu, Chi Zhang, Yinan Li, Chi Wang, Badrish Chandramouli, Johannes Gehrke, Donald Kossmann, David Lomet, 2020
https://scholar.google.com/scholar?q=ALEX%3A+An+Updatable+Adaptive+Learned+Index
4. Let's Talk About Storage & Recovery Methods for Non-Volatile Memory Database Systems — Joy Arulraj, Andrew Pavlo, Subramanya R. Dulloor, 2015
https://scholar.google.com/scholar?q=Let%27s+Talk+About+Storage+%26+Recovery+Methods+for+Non-Volatile+Memory+Database+Systems
5. An Empirical Guide to the Behavior and Use of Scalable Persistent Memory — Jian Yang, Juno Kim, Morteza Hoseinzadeh, Joseph Izraelevitz, Steve Swanson, 2020
https://scholar.google.com/scholar?q=An+Empirical+Guide+to+the+Behavior+and+Use+of+Scalable+Persistent+Memory
6. How Does Updatable Learned Index Perform on Non-Volatile Main Memory? — Leying Chen, Shimin Chen, 2021
https://scholar.google.com/scholar?q=How+Does+Updatable+Learned+Index+Perform+on+Non-Volatile+Main+Memory%3F
7. Dash: Scalable Hashing on Persistent Memory — Baotong Lu, Xiangpeng Hao, Tianzheng Wang, Eric Lo, 2020
https://scholar.google.com/scholar?q=Dash%3A+Scalable+Hashing+on+Persistent+Memory
8. FPTree: A Hybrid SCM-DRAM Persistent and Concurrent B-Tree for Storage Class Memory — Ismail Oukid, Johan Lasperas, Anisoara Nica, Thomas Willhalm, Wolfgang Lehner, 2016
https://scholar.google.com/scholar?q=FPTree%3A+A+Hybrid+SCM-DRAM+Persistent+and+Concurrent+B-Tree+for+Storage+Class+Memory
9. RECIPE: Converting Concurrent DRAM Indexes to Persistent-Memory Indexes — Se Kwon Lee, Jayashree Mohan, Sanidhya Kashyap, Taesoo Kim, Vijay Chidambaram, 2019
https://scholar.google.com/scholar?q=RECIPE%3A+Converting+Concurrent+DRAM+Indexes+to+Persistent-Memory+Indexes
Interactive Visualization: APEX: A High-Performance Learned Index on Persistent Memory