Programmatically Interpretable Reinforcement Learning: Readable Policies

This episode examines Programmatically Interpretable Reinforcement Learning (PIRL), a 2018 framework from Rice University, Google Brain, and DeepMind researchers that forces RL policies to be expressed as short, human-readable programs rather than opaque neural network weights. The discussion centers on why formal verification—proving properties like bounded steering output in a self-driving car—is tractable for small domain-specific programs but essentially impossible for networks with millions of parameters. Using the paper's driving example, the hosts unpack "policy sketches" (a switch statement branching on track position, with PID controllers filling each branch) and Neurally Directed Program Search (NDPS), which trains a conventional deep RL policy as an oracle and then searches program space to imitate its outputs via smooth regression rather than fighting a jagged, non-differentiable reward landscape. They draw out the connection to DAgger's iterative imitation-learning approach from Ross, Gordon, and Bagnell, while flagging a subtle mismatch between matching an expert's actions and matching reward through an imitation proxy. Listeners interested in AI safety, control theory, or the tension between interpretability and performance will find the concrete TORCS driving case a clear entry point into verifiable reinforcement learning.

Interactive Visualization: Programmatically Interpretable Reinforcement Learning: Readable Policies

Sources:
1. Programmatically Interpretable Reinforcement Learning — Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri, 2018
http://arxiv.org/abs/1804.02477
2. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning — Stéphane Ross, Geoffrey Gordon, J. Andrew Bagnell, 2011
https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning
3. Is Imitation Learning the Route to Humanoid Robots? — Stefan Schaal, 1999
https://scholar.google.com/scholar?q=Is+Imitation+Learning+the+Route+to+Humanoid+Robots%3F
4. ALVINN: An Autonomous Land Vehicle in a Neural Network — Dean Pomerleau, 1989
https://scholar.google.com/scholar?q=ALVINN%3A+An+Autonomous+Land+Vehicle+in+a+Neural+Network
5. Generative Adversarial Imitation Learning — Jonathan Ho, Stefano Ermon, 2016
https://scholar.google.com/scholar?q=Generative+Adversarial+Imitation+Learning
6. Verifiable Reinforcement Learning via Policy Extraction — Osbert Bastani, Yewen Pu, Armando Solar-Lezama, 2018
https://scholar.google.com/scholar?q=Verifiable+Reinforcement+Learning+via+Policy+Extraction
7. Programmatically Interpretable Reinforcement Learning — Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri, 2018
https://scholar.google.com/scholar?q=Programmatically+Interpretable+Reinforcement+Learning
8. Optimization Methods for Interpretable Differentiable Decision Trees Applied to Reinforcement Learning — Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, Sung-Hyun Son, 2020
https://scholar.google.com/scholar?q=Optimization+Methods+for+Interpretable+Differentiable+Decision+Trees+Applied+to+Reinforcement+Learning
9. Distilling a Neural Network Into a Soft Decision Tree — Nicholas Frosst, Geoffrey Hinton, 2017
https://scholar.google.com/scholar?q=Distilling+a+Neural+Network+Into+a+Soft+Decision+Tree
10. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger) — Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, 2011
https://scholar.google.com/scholar?q=A+Reduction+of+Imitation+Learning+and+Structured+Prediction+to+No-Regret+Online+Learning+%28DAgger%29
11. Continuous Control with Deep Reinforcement Learning (DDPG) — Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra, 2015
https://scholar.google.com/scholar?q=Continuous+Control+with+Deep+Reinforcement+Learning+%28DDPG%29
12. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks — Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, Mykel J. Kochenderfer, 2017
https://scholar.google.com/scholar?q=Reluplex%3A+An+Efficient+SMT+Solver+for+Verifying+Deep+Neural+Networks
13. The Sketching Approach to Program Synthesis — Armando Solar-Lezama, 2009
https://scholar.google.com/scholar?q=The+Sketching+Approach+to+Program+Synthesis
14. Syntax-Guided Synthesis (SyGuS) — Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Ghila Juniwal, Hadas Kress-Gazit, P. Madhusudan, Milo M. K. Martin, Mukund Raghothaman, Shambwaditya Saha, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, Abhishek Udupa, 2015
https://scholar.google.com/scholar?q=Syntax-Guided+Synthesis+%28SyGuS%29
Interactive Visualization: Programmatically Interpretable Reinforcement Learning: Readable Policies