1
00:00:01,000 --> 00:00:07,037
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,037 --> 00:00:51,062
[Hal Turing] Today's paper is "All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs," by Zhixiong Zhao, Zukang Xu, and colleagues at Houmo AI and the School of Integrated Circuits at Peking University. It was posted to arXiv in September 2026. It builds on "Addition is almost all you need: Compressing neural networks with double binary factorization," by Vladimír Boža and Vladimír Macko, arXiv 2025. We'll call that one DBF. The question is whether you can compress an LLM's linear weights to exactly one bit each, with no retraining, and still have something useful.

3
00:00:51,062 --> 00:01:08,152
[Dr. Ada Shannon] The claim is that earlier methods called "1-bit" aren't. Once you count everything they store, they land between about 2.5 and 3.5 bits per weight. This paper says it hits exactly 1.0, metadata included. So we should test two words in that claim: "genuine" and "usable."

4
00:01:08,152 --> 00:01:11,124
[Hal Turing] Start with why anyone wants this. Why chase one bit?

5
00:01:11,124 --> 00:01:31,697
[Dr. Ada Shannon] The paper's framing is memory traffic. A 400-billion-parameter-class model is roughly 800 gigabytes in BF16. Decoding one token at a time is mostly bandwidth-bound, because the GPU spends its time streaming weights. Fewer bits per weight means less to stream. That's the authors' argument, and it's a standard one.

6
00:01:31,697 --> 00:01:34,715
[Hal Turing] And binarization means each weight becomes plus one or minus one?

7
00:01:34,715 --> 00:02:03,833
[Dr. Ada Shannon] Times a scale. W is approximately alpha times B, where B is a sign matrix. The matmul turns into additions and subtractions. The trouble is that one sign matrix has very little capacity. Real weights have varied magnitudes and outlier channels. So methods like BiLLM from 2024, ARB-LLM, STBLLM and HBLLM add rescue machinery: extra scales, masks marking salient weights, residual passes, group assignments.

8
00:02:03,833 --> 00:02:05,830
[Hal Turing] And that machinery costs storage.

9
00:02:05,830 --> 00:02:31,929
[Dr. Ada Shannon] Yes. Effective bits per weight is total storage bits divided by weight count, including scales, masks, group parameters and outlier indices. Table 1 puts BiLLM at 2.88, ARB-LLM at 2.51, HBLLM at 3.25 and STBLLM at 3.50. AF1 claims exactly 1.0, but only for target linear weights. Embeddings, the output head and norms aren't part of that claim.

10
00:02:31,929 --> 00:02:34,716
[Hal Turing] Is there a way to get a true one bit by training instead?

11
00:02:34,716 --> 00:03:14,747
[Dr. Ada Shannon] That's the other route. Post-training quantization, PTQ, takes a finished model, uses a small calibration set, and never retrains it. Quantization-aware training, QAT, puts the low-bit constraint in the loop. OneBit and LittleBit are QAT, and they get real one-bit storage. The cost is hundreds to over a thousand GPU-hours. BitNet b1.58, by Ma and colleagues at Microsoft Research in 2024, goes further and trains ternary models from scratch. That's context, not a competitor here. AF1 wants QAT-style storage at PTQ cost.

12
00:03:14,747 --> 00:03:19,298
[Hal Turing] So how do you widen a single sign matrix's capacity without adding bits?

13
00:03:19,298 --> 00:03:45,072
[Dr. Ada Shannon] DBF's answer is to write the matrix as a product of two binary matrices. Each has its own full-precision scaling vectors around it. The inner dimension, the rank, sets the storage budget. A product of two sign matrices can take many more distinct values than one sign matrix, so the factorization buys expressiveness without extra bits per entry. Inference still needs only additions, sign flips and cheap vector scalings.

14
00:03:45,072 --> 00:03:47,487
[Hal Turing] Then the paper needs two more ideas. First one?

15
00:03:47,487 --> 00:04:09,360
[Dr. Ada Shannon] The null space. Say you approximate A times B, and B has a direction it maps to zero. An error in A that lies in that direction gets multiplied by B and vanishes from the output. So it's free to leave uncorrected. Spend your limited binary capacity on the errors that survive the product. We'll see shortly how AF1 uses this.

16
00:04:09,360 --> 00:04:10,010
[Hal Turing] Second one?

17
00:04:10,010 --> 00:04:37,410
[Dr. Ada Shannon] The Shapley value, from cooperative game theory. A component's value is its average marginal contribution across all the orders in which components join a coalition. Here the components are layers or submodules, and the payoff is how much damage compression does. Exact computation is combinatorial, so the paper samples random permutations, a Monte Carlo estimate. The point is to decide which layers deserve more rank when the total budget is fixed.

18
00:04:37,410 --> 00:04:39,453
[Hal Turing] Why not just score each layer on its own?

19
00:04:39,453 --> 00:04:55,196
[Dr. Ada Shannon] Because scoring layers one at a time ignores interactions, where two layers that are harmless alone hurt together. Averaging over orderings captures some of that. The price is many more forward passes. Whether it pays off is something we'll check against the ablation.

20
00:04:55,196 --> 00:05:06,806
[Hal Turing] Two metrics we'll need later. Perplexity is how surprised the model is by text, lower is better. The zero-shot average is mean accuracy over seven commonsense and language tasks, higher is better.

21
00:05:06,806 --> 00:05:17,487
[Dr. Ada Shannon] And chance on that scale is about 25. The crudest baseline, round-to-nearest, collapses to roughly that. So a score of 25 means the model is broken, not just worse.

22
00:05:17,487 --> 00:05:28,772
[Hal Turing] Then here's the claim as I read it. AF1 at 1.0 bits per weight outperforms existing binarization PTQ methods and is described as near-QAT.

23
00:05:28,772 --> 00:05:41,033
[Dr. Ada Shannon] Right. Hold that as a claim, not a finding. The comparison is against methods spending two to three times the bits, and we'll look at whether it's a clean comparison and whether the surviving model is actually worth running.

24
00:05:41,033 --> 00:05:43,029
[Hal Turing] Okay, so how does the method actually work?

25
00:05:43,029 --> 00:06:08,386
[Dr. Ada Shannon] The first piece is NABF, and it works as a chain with three links. The first link is inherited from DBF. Before factorizing a weight matrix, you scale it by an output-importance vector and an input-importance vector. Those come from a K-FAC-style diagonal approximation built from activations and gradients. An error on a sensitive channel then costs more in the objective, so the binary factors spend their capacity there.

26
00:06:08,386 --> 00:06:13,958
[Hal Turing] So the sensitivity weighting is already in DBF. What does AF1 add to the factorization itself?

27
00:06:13,958 --> 00:06:38,804
[Dr. Ada Shannon] The second link, and it's the new idea. DBF alternates. It fixes one factor and solves a continuous proxy for the other with ADMM. Then SVID projects that proxy onto signs times a scale. The projection leaves a residual. Call it E. But E doesn't hurt you directly. It gets multiplied by the other, fixed factor, B, so the damage is E times B.

28
00:06:38,804 --> 00:06:42,983
[Hal Turing] And some of E gets multiplied into nothing. That's the null space from earlier.

29
00:06:42,983 --> 00:07:11,265
[Dr. Ada Shannon] Right. Any part of E that B annihilates is free to leave uncorrected. So AF1 steers the correctable part of the residual toward B's approximate null space. It does this with an SVD of B times B-transpose, keeping directions whose energy falls under a one percent cutoff. Then it absorbs the correction into the existing scale vector with a closed-form least-squares fit. The sign matrices don't change, and neither does inference.

30
00:07:11,265 --> 00:07:13,773
[Hal Turing] So the fix costs no bits and adds no runtime.

31
00:07:13,773 --> 00:07:36,621
[Dr. Ada Shannon] Correct. The third link is a final pass. Freeze every sign and tune only the scale vectors, a, m and b, against the full-precision model's output distribution, using KL divergence on the calibration set. Trainable parameters per layer drop from the size of the matrices to roughly the sum of their dimensions. That is cheap, and it is still not training in the QAT sense.

32
00:07:36,621 --> 00:07:40,430
[Hal Turing] Let me pin down provenance. Which parts are DBF and which are AF1?

33
00:07:40,430 --> 00:07:55,244
[Dr. Ada Shannon] The paper says it follows DBF for the double-binary form, the ADMM-SVID solver, the K-FAC surrogate and the calibration setup. New in AF1 are the null-space compensation, the scale-only KL stage, and the allocation scheme, HiSA.

34
00:07:55,244 --> 00:07:59,656
[Hal Turing] Then HiSA. If the budget is fixed at one bit, what is left to allocate?

35
00:07:59,656 --> 00:08:15,074
[Dr. Ada Shannon] The rank. Each linear layer's intermediate rank r sets its storage. HiSA decides which layers and modules get a bigger r, and which get less, with the total pinned at 1.0 bits per weight. Nothing gets added. Capacity moves around.

36
00:08:15,074 --> 00:08:46,606
[Dr. Ada Shannon] At the layer level, each layer is a player. Degrade a layer to a proxy rank that meets the budget, and measure how much the model's negative log-likelihood rises. Sample sixty random orderings to estimate Shapley scores. Then a temperature softmax turns scores into layer budgets. Inside a layer, it scores Q, K, V, O, up, gate and down by two distortions, direction and magnitude. It normalizes them, fuses them with a Soft-OR so a weakness in either one counts, and allocates by size-aware sensitivity.

37
00:08:46,606 --> 00:08:47,814
[Hal Turing] Where does that stop helping?

38
00:08:47,814 --> 00:09:01,374
[Dr. Ada Shannon] The paper's sensitivity plots answer that. Too small a temperature piles capacity onto a few layers. Too large, and you drift toward uniform. The samples saturate around sixty at layer level and forty at module level.

39
00:09:01,374 --> 00:09:04,532
[Hal Turing] Now the results. What do the base models look like at one bit?

40
00:09:04,532 --> 00:09:33,836
[Dr. Ada Shannon] Better than the other binarizers, and clearly degraded against BF16. On LLaMA-3-8B the zero-shot average falls from about 72.5 to 45.5. That's roughly 27 points lost out of a 47-point range above chance. LLaMA-3-70B falls from 79 to 62, about 17 points. Gemma-3-12B's perplexity barely moves, 25.2 to 27.1, yet its zero-shot average still drops about 13 points.

41
00:09:33,836 --> 00:09:36,715
[Hal Turing] So perplexity can look fine while the tasks suffer.

42
00:09:36,715 --> 00:09:44,610
[Dr. Ada Shannon] Yes. The pattern is that bigger models survive better. I'd say less broken, usable at larger scale. I would not say near lossless.

43
00:09:44,610 --> 00:09:48,186
[Hal Turing] And the comparison against BiLLM, HBLLM and the rest?

44
00:09:48,186 --> 00:10:21,948
[Dr. Ada Shannon] The headline is real but mismatched. On the 70B model, HBLLM at 3.25 bits scores 53 with perplexity 9.55. AF1 at 1.0 bit scores 62 with 8.91. Beating a method with over three times the bits is a result. But the matched-budget row in the paper is AF1 at 2.5 bits, a different regime, where the 70B model scores 72.3 with perplexity 4.5. DBF shows up only as the method AF1 follows, not as a benchmarked row.

45
00:10:21,948 --> 00:10:24,362
[Hal Turing] So the gain can't be assigned to the new parts yet.

46
00:10:24,362 --> 00:10:50,740
[Dr. Ada Shannon] Not from these tables. The ablation helps a little. On a 3B model, perplexity goes from about 890 with nothing, to 130 with the surrogate, to 41 with null-space factorization, to 30 with the scale stage. HiSA then brings it to about 24. So NABF does most of the work, and HiSA's share is smaller. But it's one model, no variance, and it doesn't start from DBF as published.

47
00:10:50,740 --> 00:10:52,923
[Hal Turing] What about cost against the training methods?

48
00:10:52,923 --> 00:11:24,502
[Dr. Ada Shannon] On LLaMA-2-7B, AF1 takes 5.9 GPU-hours with perplexity 11.2. LittleBit takes 237 hours and OneBit 1,344, both landing at roughly 9 to 10. The figure labels don't make clear which value belongs to which. So the gap to QAT is real but modest, and the cost is about 40 and 228 times lower. Calibration is 128 RedPajama sequences of 2048 tokens, on RTX PRO 6000 GPUs.

49
00:11:24,502 --> 00:11:25,292
[Hal Turing] And speed?

50
00:11:25,292 --> 00:11:50,137
[Dr. Ada Shannon] Using GemLite binary kernels at batch size one, with a 1024-token prefill and 4096-token decode, an 8B model goes from 16 GB to about 1.6 GB. That's about 90 percent less memory than BF16 and 68 percent less than GPTQ 4-bit. Decoding runs about 2.5 times faster than BF16. The text doesn't state the speed against GPTQ.

51
00:11:50,137 --> 00:11:54,781
[Hal Turing] So, bottom line. Is this a usable model, or just less broken than the other binarizers?

52
00:11:54,781 --> 00:12:32,072
[Dr. Ada Shannon] Both, depending on size. Against the other binarizers it's clearly better. As a drop-in replacement for an 8B model, no. The instruction-tuned table is where that shows. It uses three harder tests. MMLU-Pro is a hard multiple-choice test of knowledge and reasoning. HumanEval asks the model to write code that passes unit tests. GSM8K is grade-school math word problems. On LLaMA-3.1-8B-Instruct, GSM8K falls from about 83 correct out of 100 to about 45. MMLU-Pro falls from 46.6 to 20.1, less than half.

53
00:12:32,072 --> 00:12:34,023
[Hal Turing] And the older method it's supposed to beat?

54
00:12:34,023 --> 00:13:04,487
[Dr. Ada Shannon] HBLLM, at about three and a quarter bits, gets 23.6 on that MMLU-Pro. So it wins there. On the 14B Qwen instruct model, HBLLM wins all three tasks. GSM8K is about 72 for HBLLM and 65 for AF1. But AF1's file is 2.9 gigabytes against 7.4. The paper calls that about nine percent lower accuracy at 39 percent of the size. That's a fair trade to describe, but it's a footprint comparison, not an equal-accuracy one.

55
00:13:04,487 --> 00:13:06,531
[Hal Turing] So what's still missing from the comparison?

56
00:13:06,531 --> 00:13:29,518
[Dr. Ada Shannon] Three controls. First, DBF at one bit with the same calibration. That isolates what null-space compensation and HiSA add. Second, a two-bit method like QuIP#, from Tseng and colleagues in 2024, AQLM from Egiazarian and colleagues, or QTIP. Those show what an extra bit buys. Third, a pruned or low-rank model at the same footprint.

57
00:13:29,518 --> 00:13:31,469
[Hal Turing] What result would settle the first one?

58
00:13:31,469 --> 00:13:54,735
[Dr. Ada Shannon] DBF at 1.0 bits per weight against AF1 on LLaMA-3-8B, same calibration data, same evaluation. If AF1 wins by a wide margin, the new pieces matter. If it's a point or two, most of the credit goes to the parent. The ablation doesn't answer that. It starts from a no-component configuration on one 3B model, not from DBF as published.

59
00:13:54,735 --> 00:13:58,543
[Hal Turing] Then the null-space step. I'm not sure it's doing what the abstraction says.

60
00:13:58,543 --> 00:14:23,946
[Dr. Ada Shannon] That's a reasonable doubt. The paper uses an approximate null space with an energy cutoff, but the text I have doesn't report how large that subspace is, or how much of the residual falls inside it. My guess is that a sign matrix may have few directions it annihilates exactly, so the correction could be a meaningful fix or close to a per-channel rescale. The ablation shows the step helps in aggregate. It doesn't show why.

61
00:14:23,946 --> 00:14:25,757
[Hal Turing] What about the claim of genuine one bit?

62
00:14:25,757 --> 00:14:58,497
[Dr. Ada Shannon] It holds for the target linear weights, which the paper states. But the 8B model comes out at 1.56 gigabytes. Eight billion bits would be about one gigabyte. My inference, not the paper's statement, is that embeddings, the output head and norms make up the difference. The paper doesn't itemize them. So the all-in size is bigger than a strict one-bit accounting suggests. That's not dishonest, but it's the same kind of scoping the paper criticizes in baselines. The KV cache and activations also aren't compressed.

63
00:14:58,497 --> 00:14:59,519
[Hal Turing] And the speedup?

64
00:14:59,519 --> 00:15:18,327
[Dr. Ada Shannon] It's 2.5 times over BF16, at batch size one, on one kernel library, three models, decoding only. Batch one is memory-bound, which is the best case for a small weight file. Large-batch and long-context serving weren't tested, and the text doesn't give speed against GPTQ four-bit.

65
00:15:18,327 --> 00:15:19,813
[Hal Turing] What about the evidence itself?

66
00:15:19,813 --> 00:15:50,092
[Dr. Ada Shannon] Single runs, no variance. One ablation model. Two instruct models. And the QAT perplexity labels in the time figure are unclear. So I wouldn't claim anything about statistical significance. The gap to the training-based methods looks modest on perplexity, roughly 11 versus 9 to 10 on LLaMA-2-7B. But the instruct results show reasoning and code losing a lot more than that suggests. Near-QAT is a perplexity claim on one model.

67
00:15:50,092 --> 00:15:55,897
[Hal Turing] Another gap. The signs are frozen and only the scales are tuned. What happens when someone wants to fine-tune?

68
00:15:55,897 --> 00:16:09,457
[Dr. Ada Shannon] The paper treats the compressed model as a final artifact. It gives no story for LoRA-style adaptation or editing, and any change to the factors would presumably undo the compensation. A light fine-tune of the scales is a natural follow-up.

69
00:16:09,457 --> 00:16:12,940
[Hal Turing] So where would you actually use a 1.6 gigabyte 8B model?

70
00:16:12,940 --> 00:16:38,947
[Dr. Ada Shannon] This is our reading, not the paper's. Memory-limited devices where some quality loss is acceptable. A draft model for speculative decoding, or a cheap routing model. And a bigger question: does a 70B-class model at one bit beat a smaller model at higher precision? That comparison wasn't run, and it matters. No MoE or long-context tests appear either. Combining this with KV-cache compression is the obvious next step.

71
00:16:38,947 --> 00:17:05,046
[Hal Turing] So, to sum up. AF1 shows that post-training binarization can reach a true 1.0 bits per weight on the linear layers, in hours rather than hundreds of GPU-hours, with far less loss than earlier binarizers. The boundary is that what survives depends on model size and task, and the control comparison against DBF at one bit is missing.

72
00:17:05,046 --> 00:17:08,111
[Dr. Ada Shannon] That's the right summary. Thanks for listening, everyone.

73
00:17:08,111 --> 00:17:09,643
[Hal Turing] Thanks for listening. Goodbye.