AI Post Transformers // Episode Companion

Deep Drone Acrobatics: Vision-Only Zero-Shot Sim-to-Real Flight

Kaufmann, Loquercio, Ranftl, Müller, Koltun, Scaramuzza RSS 2020 UZH · ETH Zürich · Intel ISL Paper PDF ↗

System Pipeline

Three asynchronous input branches feed a shared policy network running at 100 Hz onboard the quadrotor.
Feature-track branch (vision) IMU branch Reference-trajectory branch

Three Maneuvers

Click a maneuver to trace its flight path. Constant tangential velocity, 7th-order polynomial transitions.

DAgger Imitation Loop — Not Reinforcement Learning

A privileged MPC expert with ground-truth state labels the actions a vision-limited student should have taken at the states it actually visited.

Tracking Error Across DAgger Iterations

150 rollouts total, aggregated in batches of 30 — five retraining iterations. Mock convergence curve illustrating the aggregation effect.

Input Abstraction: Raw Pixels vs. Feature Tracks

Toggle the observation type fed to the policy. Feature tracks (Harris corners + Lucas-Kanade, epipolar-filtered) compress the sim/real gap; raw pixels don't.

Tracking Error: Simulation Comparison

Average position-tracking error, meters below — the only setting where exact ground truth exists to compute it at all.

Success Rate Matrix

Hover a cell for details. Rows: method. Columns: maneuver / test condition (simulation).

Real-World Hardware Runs (n=10 per maneuver)

Pilot-judged binary success — no ground-truth tracking error is computable outdoors.

Intellectual Lineage

The privileged-expert framework and abstraction argument were both imported, not invented here.

Open Questions

Combo maneuver drop — the only long-horizon compounding test falls to 95% for the full model and collapses to 0% for both ablations, with no diagnosis of why.
Subjective success metric — real-world "success" is a single safety pilot's judgment call, with no stated threshold or second observer.
Vision never randomized — IMU bias and thrust-to-weight are randomized ±10%, but scene geometry, lighting, and texture are not — despite flying in one fixed real-world room.
Generalization overreach — the conclusion claims applicability "beyond autonomous flight," from one 1.15 kg platform in one space.

References