Programmatically Interpretable Reinforcement Learning: Readable Policies

⟶ arXiv:1804.02477 ICML 2018 Rice University · Google Brain · DeepMind Verma, Murali, Singh, Kohli, Chaudhuri

PIRL forces an RL policy to be a short, human-readable program instead of a neural network — trading raw capacity for something a theorem prover can actually reason about. This page visualizes the policy-sketch structure, the Neurally Directed Program Search (NDPS) loop, and the ablation/robustness numbers the hosts pushed back on.

From weights to readable code

Instead of post-hoc explanations glued onto a black box, PIRL restricts the policy itself to a small domain-specific program. The pipeline below is the whole framework in one diagram: an oracle network is trained normally, then a program is searched for that imitates it — and only the program ever gets deployed or verified.

Why verification chokes on networks but not programs

Toggle between the two artifacts a verifier has to reason about. Each cell in the grid stands in for a parameter a solver must track — the neural policy's DDPG network has three 600-unit layers, dwarfing what tools like Reluplex could handle in 2017 (~300 nodes). The synthesized program has a handful of numeric gains and one branch condition.

low magnitude mid high magnitude

Figure 2's structural prior: switch on TrackPos

The sketch fixes the shape — a branch on how far the car sits from track center, PID control inside each branch — and leaves only the condition threshold and the PID gains as unknowns for the search to fill in. Hover a branch to see its role.

Does the branch earn its keep?

Compare the full branching sketch against NoIF — a single fixed PID controller with no branching at all. On reward, the paper's own ablation shows them essentially tied.

Neurally Directed Program Search

NDPS never searches for reward directly — it freezes a trained DDPG oracle and turns synthesis into a sequence of smooth regression problems against that oracle's outputs, refreshing the state set DAgger-style as the program improves.

Ablation reward — ties, not wins

Naive, NoAug, and NoSketch time out entirely on both tracks within the 12-hour budget. NoIF and full NDPS are within rounding error of each other.

Steering smoothness

NDPS steering std-dev is 4–5× lower than DDPG's — plausibly because DDPG had no jerk penalty, not because programs are inherently smoother.

Sensor-dropout survival (meters)

Toggle the blocking probability. NDPS survives far longer — on sensors the program was structurally built to tolerate.

Transfer to unseen tracks

DRL crashes on every transfer track; NDPS completes all four. A dramatic result — and, per the critique, a suspiciously total one.

Verification scale — where the paper is on firm ground

Log-scale node counts. Reluplex (2017) tops out around 300 nodes; the DDPG oracle has roughly 1,800 across three layers; the synthesized program is small enough for symbolic execution to bound directly.

Classic-control appendix vs. the abstract's framing

CartPole and Acrobot scores against their standard "solved" thresholds. Two of three land below the bar.