FastPersist: Making Per-Iteration Deep Learning Checkpointing Nearly Free

References

1. FastPersist: Accelerating Model Checkpointing in Deep Learning — Wang, Ruwase, Xie, He, 2024 (arXiv:2406.13768)
2. CheckFreq: Frequent, Fine-Grained DNN Checkpointing — Mohan, Phanishayee, Chidambaram, 2021
3. Gemini: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints — Wang et al., 2023
4. Check-N-Run — Eisenman et al., 2022
5. ByteCheckpoint — Wan et al., 2024
6. ZeRO-Infinity — Rajbhandari, Ruwase, Rasley, Smith, He, 2021
7. Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training — Zhao et al., 2022
8. What Modern NVMe Storage Can Do, And How To Exploit It — Haas, Leis, 2023
9. PyTorch Distributed Checkpoint (DCP) and async_save — PyTorch team, 2023–2024
10. OPT-175B logbook and Llama 3 herd paper (failure analyses) — Zhang et al.; Llama Team, 2022; 2024
11. Young/Daly optimal checkpoint interval; multi-level checkpointing (SCR, FTI) — Young; Daly; Moody et al.; Bautista-Gomez et al.
12. GPUDirect Storage / cuFile and SPIN — NVIDIA; Bergman et al.
13. ReCycle / Oobleck / Bamboo — resilient training without checkpoint restart, 2023–2024