Approaching Shannon Bound: Lossless LLM Weight Compression

Hongshi Tan, Yao Chen, Gustavo Alonso, Weng-Fai Wong, Bingsheng He · NUS / HUST / ETH Zurich · Jun 2026 arXiv:2606.15789 ↗ Interactive Viz ↗

Weights stored in bf16 carry less real information than 16 bits implies. Entropy measurements across six models and seven numeric formats reveal gaps of several bits per weight — recoverable losslessly via Asymmetric Numeral Systems (ANS), fast enough to decode tile-by-tile inside a live GPU kernel. The headline "up to 10x" and the demonstrated 1.2–1.6x end-to-end throughput win describe two different regimes.

References