Deep Drone Acrobatics: Vision-Only Zero-Shot Sim-to-Real Flight

This episode explores "Deep Drone Acrobatics," which trains a quadrotor to fly extreme maneuvers — a Power Loop, Barrel Roll, and Matty Flip — using only an onboard camera and IMU, with no external motion capture. The discussion centers on how the policy is trained entirely in simulation via DAgger imitation learning, where a privileged model-predictive controller with perfect ground-truth state acts as an expert that a vision-limited student imitates, rather than through reinforcement learning or reward shaping. A key focus is the sim-to-real gap: at high accelerations, motion blur degrades vision-based state estimation, so the paper's "input abstraction" approach feeds the network geometry-based feature tracks instead of raw pixels, drawing on prior work showing that shared abstractions between simulated and real observations shrink the performance gap. The conversation also traces the paper's intellectual lineage, connecting it to "Does Computer Vision Matter for Action?" and "Learning by Cheating," while highlighting why acrobatic flight is a harder version of the sim-to-real problem than driving, since a flipping drone has no margin for hesitation. Listeners interested in robotics, sim-to-real transfer, or imitation learning will find a concrete, technically grounded case study of zero-shot policy transfer under extreme physical constraints.