1
00:00:01,000 --> 00:00:08,709
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,709 --> 00:00:44,514
[Hal Turing] Today's paper is "The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction." It's from Xiaoyu Li and colleagues Zhizhou Sha and Chiwun Yang, at the University of New South Wales, the University of Texas at Austin, and City University of Hong Kong. It's a preprint dated October 2026. The claim is that the extra width DeepSeek bought with mHC fades with depth, and that the training dynamics get sticky near permutation matrices.

3
00:00:44,514 --> 00:00:55,891
[Dr. Ada Shannon] Right, and this is a theory paper. It trains no models. It tries to explain two odd observations from DeepSeek's own mHC paper, and the answer says something about what that extra width is worth.

4
00:00:55,891 --> 00:01:08,384
[Hal Turing] Then let's state the question once. If every mixer is a doubly stochastic matrix, what does that constraint give you, what does it cost, and how does gradient descent move a mixer around the space of those matrices?

5
00:01:08,384 --> 00:01:29,885
[Dr. Ada Shannon] Yes. For the background, start with the residual connection. In the ResNet paper, He and colleagues at Microsoft Research, 2016, a layer computes x plus F of x. The earlier signal reaches later layers unchanged, forward and backward. That identity path is why deep networks train at all.

6
00:01:29,885 --> 00:01:31,604
[Hal Turing] And hyper-connections widen that path.

7
00:01:31,604 --> 00:02:01,511
[Dr. Ada Shannon] Zhu and colleagues, ICLR 2025. The residual stream becomes n parallel streams, and n is 4 in mHC. Each layer reads from the streams, writes back to them, and mixes them with an n-by-n matrix. Nothing constrains that matrix, so it can amplify or attenuate without bound. The mHC authors, Xie and colleagues at DeepSeek-AI, report composite gains on the order of a thousand and loss spikes at 27 billion parameters. The identity path was gone.

8
00:02:01,511 --> 00:02:03,601
[Hal Turing] So mHC's fix was to constrain the mixer.

9
00:02:03,601 --> 00:02:23,570
[Dr. Ada Shannon] To doubly stochastic: nonnegative entries, every row and every column summing to 1. The set of all such matrices is the Birkhoff polytope. Its corners are exactly the permutation matrices, and the uniform matrix, every entry 1 over n, sits in the middle. Every doubly stochastic matrix is a blend of permutations.

10
00:02:23,570 --> 00:02:25,938
[Hal Turing] How do you get a matrix like that out of a network?

11
00:02:25,938 --> 00:02:48,740
[Dr. Ada Shannon] Sinkhorn normalization, from Sinkhorn and Knopp in 1967. Take a matrix of logits W and exponentiate it. Then alternately normalize the columns and the rows. Think of a softmax applied over rows and then over columns, back and forth. mHC runs 20 sweeps and stops, ending on a row step. So rows sum to 1 exactly, and columns only approximately.

12
00:02:48,740 --> 00:02:49,948
[Hal Turing] So what are the puzzles?

13
00:02:49,948 --> 00:03:19,251
[Dr. Ada Shannon] Two, both from the mHC paper. First, multiply the mixers across 30, 60 or 90 layers and the composite looks almost uniform. Entries sit between about 0.17 and 0.35, while single layers range from 0.05 to 0.5. Second, the composite backward gain drifts to about 1.6, so it isn't exactly 1. A third observation comes from Guo and colleagues, who found the composite of 24 trained mixers is close to rank one.

14
00:03:19,251 --> 00:03:24,453
[Hal Turing] A near-uniform composite is rank one. That means the streams have been averaged together.

15
00:03:24,453 --> 00:03:30,118
[Dr. Ada Shannon] That's the worry. If the extra streams get averaged away over depth, what is the extra width buying?

16
00:03:30,118 --> 00:03:31,512
[Hal Turing] Why does this matter right now?

17
00:03:31,512 --> 00:03:50,645
[Dr. Ada Shannon] The paper cites DeepSeek-V4 as using mHC. I couldn't check that outside the paper, so take it as their citation. A lot of follow-ups also swap out Sinkhorn. One example is mHC-lite by Yang and Gao, 2026, titled "You don't need 20 Sinkhorn-Knopp iterations."

18
00:03:50,645 --> 00:03:52,642
[Hal Turing] So is there a thread through all of this?

19
00:03:52,642 --> 00:04:10,800
[Dr. Ada Shannon] One number: the second singular value of a mixer, sigma two. It governs how well signal propagates, how fast learning moves near the corners, and how expensive exact projection is. The four claims are the two-channel split, the Fisher-Rao geometry, the sticky vertices, and the cost of truncated Sinkhorn.

20
00:04:10,800 --> 00:04:14,979
[Hal Turing] Let's start with the first claim. What does the constraint actually give you?

21
00:04:14,979 --> 00:04:38,199
[Dr. Ada Shannon] One exact guarantee. Split the four streams into their average and the deviations from that average. A doubly stochastic mixer fixes the all-ones vector, so the average passes through with gain exactly one, forward and backward. On the average, mHC is an ordinary residual network. Everything the extra width adds lives in the deviations, the difference channel.

22
00:04:38,199 --> 00:04:40,103
[Hal Turing] And the difference channel isn't so lucky.

23
00:04:40,103 --> 00:05:19,717
[Dr. Ada Shannon] Each layer shrinks it by the mixer's second singular value, sigma two. The bound is sigma two at most one minus n times the smallest entry. It's exact for a blend of one permutation and the uniform matrix. Compose layers and the shrinkage multiplies. The paper's worked example assumes n of 4 and a smallest entry of 0.02 in every layer. Then each layer shrinks the differences by at most 0.92, a memory of about twelve layers. Over thirty layers the shrinkage compounds to below 0.091. That 0.02 is an assumption, not a measurement.

24
00:05:19,717 --> 00:05:25,104
[Hal Turing] So the horizon, one over one minus sigma two, is how many layers the extra streams remember.

25
00:05:25,104 --> 00:05:36,156
[Dr. Ada Shannon] Yes. And there's a rigidity result. Among nonnegative mixers, only permutations avoid shrinking the differences. Anything that nearly preserves them must sit near a corner.

26
00:05:36,156 --> 00:05:40,104
[Hal Turing] Which brings in the training dynamics. How do you move toward a corner?

27
00:05:40,104 --> 00:06:01,002
[Dr. Ada Shannon] Through the logits. Vertices sit at infinity in logit space, because a permutation needs zeros and exp of a finite number is never zero. The key lemma says backprop through Sinkhorn computes exactly the Fisher-Rao gradient. Fisher-Rao measures distance so that small entries are cheap to change in relative terms. A tiny entry can double without much cost.

28
00:06:01,002 --> 00:06:05,274
[Hal Turing] But training applies that gradient to the logits, not to the matrix itself.

29
00:06:05,274 --> 00:06:36,853
[Dr. Ada Shannon] That's the catch. The metric gets applied twice, once in the gradient and once in the map from logits to matrix. Plain logit descent follows a squared Fisher-Rao metric. The straight-through update skips one application, by treating the gradient with respect to H as the gradient for the logits. That version is exactly entropic mirror descent. It's also exponentiated gradient, and Riemannian descent with the Sinkhorn retraction. One update, three names.

30
00:06:36,853 --> 00:06:38,850
[Hal Turing] What does the double application cost you?

31
00:06:38,850 --> 00:07:12,148
[Dr. Ada Shannon] Viscosity. Let epsilon be the distance to the nearest permutation. The logit gradient is proportional to epsilon, so it vanishes at every corner, for every loss. The constant in their bound, 4n cubed, isn't sharp by their own account. The two-stream case shows it cleanly. Under Fisher-Rao, the off-diagonal entry q shrinks at a rate proportional to q times one minus q, which is exponential decay. Under logit flow the rate is proportional to q squared, so q only falls like one over time.

32
00:07:12,148 --> 00:07:13,959
[Hal Turing] And that bites in both directions.

33
00:07:13,959 --> 00:07:39,269
[Dr. Ada Shannon] Yes, approaching and leaving. Combine it with the memory result. A long horizon forces at least half the layers near a vertex. Reaching that configuration costs time linear in depth for logit flow, and logarithmic in depth for Fisher-Rao. The paper's own qualification is that AdamW divides out the epsilon-sized scale, so Adam moves exponentially anyway. The viscosity claim is about gradient flow and the landscape.

34
00:07:39,269 --> 00:07:42,241
[Hal Turing] Then there's the fourth claim, the Sinkhorn budget.

35
00:07:42,241 --> 00:08:04,067
[Dr. Ada Shannon] Near the answer, each sweep shrinks the error by sigma two squared. That's the same quantity as the horizon. So the long-memory mixers are exactly the ones Sinkhorn projects slowest. Column-sum errors also multiply across layers into the backward-gain drift. Their heuristic says twenty sweeps over sixty layers limits the horizon to about eight layers, and they call it a heuristic.

36
00:08:04,067 --> 00:08:06,111
[Hal Turing] Is there an escape from nonnegativity?

37
00:08:06,111 --> 00:08:32,117
[Dr. Ada Shannon] Isometric mixers. Those that fix the ones vector and keep difference norms form a group isomorphic to O of n minus one. No boundary, no viscosity, and a Cayley retraction needs one linear solve. The price is negative entries, so the mix is no longer a convex combination. Guo and colleagues use full SO(n) mixers, which couple the mean and differences, so the mean no longer has unit gain.

38
00:08:32,117 --> 00:08:33,928
[Hal Turing] What did the numerical checks show?

39
00:08:33,928 --> 00:09:12,706
[Dr. Ada Shannon] These are checks of the theorems, not benchmarks of a model. First, random mixers at n of 4. At logit scale 4, the per-layer factor is about 0.55, so a memory of about two layers. Even sharp random mixers forget quickly. Second, the approach to a corner. Logit descent decays as one over t, with a fitted slope of minus 1.05, while straight-through decays exponentially. In the three-stream picture, straight-through gets within 0.003 of the identity by time 8. Logit flow needs roughly 400 units to get within 0.001. That's about fifty times longer, though the targets differ, so it's only a rough illustration.

40
00:09:12,706 --> 00:09:13,820
[Hal Turing] And the sweeps?

41
00:09:13,820 --> 00:09:30,585
[Dr. Ada Shannon] A mixer with a ten-layer memory needs 23 sweeps to get column sums accurate to one in a thousand. mHC stops at 20. A fifty-layer memory needs about 81, roughly four times the budget. The need grows about 1.5 sweeps per layer of horizon.

42
00:09:30,585 --> 00:09:34,904
[Hal Turing] We've got theorems and a set of numerical checks. What do those checks establish?

43
00:09:34,904 --> 00:09:59,378
[Dr. Ada Shannon] That the numerics agree with theorems that were already proved. That's all. Nothing here is a trained model. Every experiment is four streams on a laptop CPU, with random logits or a hand-built linear loss, and exact projections instead of the twenty-sweep truncation mHC uses. So "experiments confirm the predicted rates" means the simulations match the math.

44
00:09:59,378 --> 00:10:04,904
[Hal Turing] So fading memory, sticky vertices, gain drift: none of them is observed in a real mHC.

45
00:10:04,904 --> 00:10:17,861
[Dr. Ada Shannon] Right. A decisive test would be a small trained mHC with measured composite second singular value, vertex proximity over training, and backward gain. The paper offers sigma two as a diagnostic and never applies it.

46
00:10:17,861 --> 00:10:21,437
[Hal Turing] But it does explain something real, the near-uniform composites.

47
00:10:21,437 --> 00:10:48,929
[Dr. Ada Shannon] The only real observations it explains are the two in Xie and colleagues' mHC paper: the drift near 1.6 and the near-uniform composites. Neither is re-measured. And the explanation is consistent with the data, not predictive. The bound forces collapse only if you assume a minimum entry per layer, and the worked example assumes it. Trained mixers have entries as small as 0.05 or less, and when an entry is near zero the bound says nothing.

48
00:10:48,929 --> 00:10:58,542
[Hal Turing] Is that bounded memory even a defect? Each layer writes fresh differences. Maybe the last few layers plus a lossless mean path is all the extra streams need.

49
00:10:58,542 --> 00:11:26,499
[Dr. Ada Shannon] That's my reading too. The paper never shows the horizon is too short for any task. The blocks and read-write gates adapt to whatever the mixers do, and the paper holds them fixed. Causal work like Ablate and Rescue, by William Peng and colleagues, 2026, and Pengxiang Zhao's study of mHC's near-identity mixing are where to look for whether trained models use the difference channel at all. Near-identity mixers would sit near vertices and contradict the uniform picture.

50
00:11:26,499 --> 00:11:31,004
[Hal Turing] Then the optimizer. Viscosity is a statement about plain gradient flow.

51
00:11:31,004 --> 00:12:02,351
[Dr. Ada Shannon] And mHC trains with AdamW. Remark 5.7 argues that Adam divides out the small gradient scale, so it escapes exponentially. That argument ignores the epsilon in the denominator, minibatch noise, clipping, schedules, and the fact that the logits are a learned projection of the input, not free parameters. The paper's own toy run agrees with the remark, which isn't independent support. The authors list it as open problem one. Once Adam is exempt, how much the flagship result matters for real training is unresolved.

52
00:12:02,351 --> 00:12:04,348
[Hal Turing] What about the input-dependent logits?

53
00:12:04,348 --> 00:12:23,388
[Dr. Ada Shannon] The flow holds input by input, up to a weighting. But the gradient is averaged over thousands of tokens that want different mixers. Whether viscosity survives that is unknown. And the toy loss has one optimal permutation, which is the best case for the exponential rate. If the real optimum is interior, vertices may not matter.

54
00:12:23,388 --> 00:12:27,382
[Hal Turing] The straight-through claim says no backward pass through Sinkhorn is needed.

55
00:12:27,382 --> 00:13:07,041
[Dr. Ada Shannon] Under exact projection. With twenty sweeps, column sums are off, so the identity is approximate. Nobody has compared straight-through against full backprop at scale. The memory saving is also asserted, not measured. The reference there is Douik and Hassibi, 2019, on Sinkhorn retraction geometry, plus Helmbold and Warmuth's 2009 permutation learning with exponential weights. Both already contain pieces of it, so the three-way identity is neat but arguably incremental. Same for the rate: Knight, 2008, gave the sigma-two-squared sweep rate, and Mei and colleagues, 2020, the simplex version of the one-over-t result.

56
00:13:07,041 --> 00:13:09,131
[Hal Turing] And the eight-layer ceiling from truncation?

57
00:13:09,131 --> 00:13:39,271
[Dr. Ada Shannon] It's a heuristic fitted to one number, the drift at sixty layers, assuming an initial error of about one. A different run could move it. Other drift sources, like gate scaling or kernel precision, aren't ablated. And since random mixers forget in about two layers, the ceiling may never bind. mHC-lite replaces Sinkhorn with an exact mix of permutations, so it's a direct test. If it matches mHC at equal quality, the truncation section matters only modestly in practice. I can't say more, because I don't have its numbers.

58
00:13:39,271 --> 00:13:42,521
[Hal Turing] Does collapse hurt quality? Orthogonal designs avoid it.

59
00:13:42,521 --> 00:14:07,506
[Dr. Ada Shannon] Collapse being bad is a premise here. oHC by Haoqiang Guo and colleagues, and JPmHC by Biswa Sengupta and colleagues, are the places to check, and the paper doesn't tabulate them. One more thing worth saying plainly: the authors disclose that AI helped draft and check proofs and write the experiment code, and they say they verified both. That's a reason to want an independent trained-model replication.

60
00:14:07,506 --> 00:14:14,426
[Hal Turing] So if I'm running a lab that trains with mHC, what do I take from this? Start with what the paper actually nailed down.

61
00:14:14,426 --> 00:14:36,996
[Dr. Ada Shannon] Four things are solid, and all four are about the idealized mHC. The mean channel has gain exactly one for any doubly stochastic mixer. The difference channel's memory has a rigorous upper bound. The straight-through update is exactly entropic mirror descent. And long memory and fast Sinkhorn convergence pull against each other, because the same second singular value sets both.

62
00:14:36,996 --> 00:14:39,689
[Hal Turing] And what can someone check on a trained model this week?

63
00:14:39,689 --> 00:15:06,253
[Dr. Ada Shannon] Two cheap diagnostics, both suggested by the paper and neither applied by it. First, compute the second singular value of every layer's mixer. That gives you the horizon per layer, and the product across layers tells you how much of the extra width survives. Second, after the truncated Sinkhorn loop, look at the final column-sum error. Proposition 6.2 ties that error to the drift in backward gain, so it works as a bound you can read off directly.

64
00:15:06,253 --> 00:15:09,364
[Hal Turing] Those are measurements on a checkpoint, not a new training run.

65
00:15:09,364 --> 00:15:17,677
[Dr. Ada Shannon] Right, which is why they're a good first step. They would also tell us whether the worst-case bound we worried about earlier is anywhere near what trained mixers do.

66
00:15:17,677 --> 00:15:21,624
[Hal Turing] What about design? Does the paper push anyone to change the architecture?

67
00:15:21,624 --> 00:15:48,745
[Dr. Ada Shannon] It draws consequences, and they come in three groups. The straight-through update needs no backward pass through Sinkhorn, and it is exact mirror descent, as long as you ignore the truncation. Then there are exact parameterizations that remove the truncation limit: mHC-lite, KromHC, TBP-mHC, go-mHC and TEMPER. Wang and Qiu take another route, projecting with implicit differentiation instead of unrolling the loop.

68
00:15:48,745 --> 00:15:50,742
[Hal Turing] And the third group is the orthogonal one.

69
00:15:50,742 --> 00:16:11,269
[Dr. Ada Shannon] Yes. JPmHC, the spectral-sphere variants and oHC escape collapse by giving up nonnegativity. Among those, Guo et al.'s evidence currently favors the full orthogonal group over the mean-preserving subgroup the paper analyzes. That is a small empirical result, though, and the paper doesn't settle it.

70
00:16:11,269 --> 00:16:12,987
[Hal Turing] So where does the theory go next?

71
00:16:12,987 --> 00:16:40,201
[Dr. Ada Shannon] Appendix G lists several. The first is rate theorems for Adam-type optimizers on the polytope, since that is what people actually run. Then escape-rate theory for more than two streams, where the clean closed forms stop. The Lyapunov exponent of random products with input-dependent mixers is a better effective horizon than the worst-case bound. There are also metrics induced by the exact parameterizations, and the question of which isometric inductive bias is better.

72
00:16:40,201 --> 00:16:42,662
[Hal Turing] If you could only run one experiment, which one?

73
00:16:42,662 --> 00:17:13,452
[Dr. Ada Shannon] Train a small mHC model twice, with Sinkhorn and with an exact parameterization like mHC-lite. Log the composite second singular value, the horizon, how close mixers sit to vertices, and the backward gain throughout training. That one run tests the fading memory, the stickiness and the truncation claim on a real model. If the two versions behave alike, the truncation section matters less than the paper suggests. If they diverge, that would be the first real-model evidence for the theory.

74
00:17:13,452 --> 00:17:39,133
[Hal Turing] Then here's where I land. The paper gives a clean account of what the doubly stochastic constraint does and where it gets expensive: exact on the mean channel, bounded on the differences, and a real tension with Sinkhorn's budget. The boundary is that it's all theory of the idealized design, with no trained-model check. Until someone measures those mixers in a real model, the horizon claims stay a hypothesis.

75
00:17:39,133 --> 00:17:40,433
[Dr. Ada Shannon] That's a fair summary.

76
00:17:40,433 --> 00:17:43,359
[Hal Turing] Thanks for listening, everyone. And thank you, Ada.

77
00:17:43,359 --> 00:17:45,402
[Dr. Ada Shannon] Thanks, Hal. Goodbye, everyone.