1
00:00:01,000 --> 00:00:44,625
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.
Today we're digging into a paper called DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. It's from Julien Siems et al. — six co-authors total — out of the University of Freiburg, the ELLIS Institute Tübingen, Microsoft Research, Prior Labs, Istituto Italiano di Tecnologia, and University College London. It's accepted at NeurIPS 2025, posted to arXiv earlier this year. Ada, this one's got a name that sounds like a household cleaning product, but the math inside is anything but mundane.

2
00:00:44,625 --> 00:01:21,250
[Dr. Ada Shannon] It really does read like something you'd find on a supermarket shelf. But honestly, Hal, what sold me on this paper wasn't the benchmark charts — it's that they hand you an actual theoretical reason the method should work before they ever run an experiment. That's rarer than it should be in this field. A lot of linear-RNN papers show you a plot and say trust us, it scales. This one starts from a geometric fact about how reflections compose, builds a real theorem out of it, and only then goes and tests whether reality agrees. That ordering is what makes me take the empirical results seriously instead of squinting at them wondering what's cherry-picked.

3
00:01:21,250 --> 00:01:46,050
[Hal Turing] Okay, before we get into the geometry, let's back up for anyone who hasn't touched this corner of the field. What do you mean by 'state-tracking,' and why is it something transformers, and even newer models like Mamba, supposedly struggle with? Every few months there's some paper claiming a fundamental limitation of transformers, and it's genuinely hard to know which of those are real math versus which are just measuring the wrong benchmark.

4
00:01:46,050 --> 00:02:41,450
[Dr. Ada Shannon] Fair skepticism. Picture a shell game — a few cups, one ball, someone keeps swapping pairs of cups. State-tracking is just: can the model tell you where the ball ends up after a long sequence of swaps? Formally that's composing elements of a permutation group, and the simplest version is S2, tracking parity — whether an even or odd number of swaps happened. Hao, Angluin, and Frandsen showed transformers can't solve arbitrary-length parity in finite precision, and Merrill and Sabharwal's result on the circuit-complexity classes TC0 versus NC1 gives the structural reason why — self-attention sits in a class that can't express certain sequential state updates no matter how you scale it. Diagonal linear RNNs like Mamba and GLA inherit the same ceiling, because a diagonal state-transition matrix only scales each channel independently — it can never mix channels the way tracking a permutation requires.

5
00:02:41,450 --> 00:02:59,450
[Hal Turing] So diagonal matrices are stuck in a lower complexity class permanently, no matter how much data or compute you throw at it — that's a much stronger claim than 'this architecture underperforms.' So DeltaNet came along promising something better. What's actually different about—

6
00:02:59,450 --> 00:03:36,550
[Dr. Ada Shannon] Oh, wait, wait — let me jump in, because this is the part people gloss over. DeltaNet's recurrence isn't just a better RNN update bolted on ad hoc — it can be read as one step of online gradient descent per token, minimizing a squared error on an associative recall objective. And when you write out algebraically what that gradient step produces, the state-transition matrix is exactly a generalized Householder reflection — identity minus beta times a key vector times its own transpose. That's not a coincidence. That's the seed the entire paper grows out of.

7
00:03:36,550 --> 00:03:55,350
[Hal Turing] That reframing — recurrence as gradient descent — is honestly one of the more elegant things I've read in this space in a while. So if DeltaNet is one gradient step per token, giving you one Householder reflection, I'm guessing DeltaProduct's whole pitch is: what if you just took more steps?

8
00:03:55,350 --> 00:04:29,125
[Dr. Ada Shannon] Exactly right, and it's almost suspiciously simple once you see it. Take n_h gradient steps per token instead of one, using extra key-value pairs, and your state-transition matrix becomes a product of n_h generalized Householder reflections instead of just one. At n_h equals one you're back to plain DeltaNet — diagonal plus rank-one. Crank n_h up and you interpolate toward a dense, fully expressive matrix, at the cost of more compute per token. It's a genuine dial, not a binary switch — you choose your point on the expressivity-efficiency trade-off instead of being stuck with whatever structure your architecture happened to ship with.

9
00:04:29,125 --> 00:04:49,600
[Hal Turing] And here's where the geometry actually gets fun, right? You've mentioned before that composing two reflections doesn't just give you a bigger reflection — it gives you a rotation. That feels like it should matter a lot for why two steps buys you more than naive intuition suggests. Is that the Cartan-Dieudonné thing you keep bringing up?

10
00:04:49,600 --> 00:05:29,600
[Dr. Ada Shannon] Right — the Cartan–Dieudonné theorem: any orthogonal transformation can be built from a bounded number of reflections, and two reflections composed together already give you a full rotation, not just a bigger reflection. So going from n_h equals one to n_h equals two isn't a small increment, it's a qualitative jump in what a single recurrence step can represent. And that composition happens before anything else touches the state — no nonlinearity, no normalization, no readout sits between reflection one and reflection two. Stack two separate layers at n_h equals one instead, and there's a readout and a fresh linear map sandwiched in between, so the reflections never get to compose that cleanly.

11
00:05:29,600 --> 00:05:56,075
[Hal Turing] Hold on, I actually disagree with you there, Ada. Isn't that a distinction without much of a difference? Stack enough layers and a deep network is a universal approximator anyway — surely with enough depth at n_h equals one you can approximate whatever composed rotation you need. Why does it matter whether the composition happens inside one recurrence step versus across depth, if the end result is the same expressive power in principle?

12
00:05:56,075 --> 00:06:33,050
[Dr. Ada Shannon] That's not just a hand-wave, though — it's a structural difference in the computational graph, and we'll see it show up concretely once we get to the experiments. Depth gives you expressivity too, eventually, nobody's disputing that a deep enough network can approximate almost anything given enough training. The claim here is narrower and sharper: for this family of state-tracking problems, increasing n_h gets you there in far fewer layers than increasing depth at n_h equals one does, because the composition inside one step is exact, not something the network has to learn to approximate across layers.

13
00:06:33,050 --> 00:06:58,325
[Hal Turing] Okay, but 'the theory says it's exact' still feels like it could be an artifact of how they set up the proof, not necessarily how training actually behaves — bounds like that are notoriously loose in practice, and plenty of 'impossible in theory' results get sidestepped empirically by scale or clever initialization. I want to see it actually hold up on real training runs before I fully buy the depth-can't-substitute story.

14
00:06:58,325 --> 00:07:18,975
[Dr. Ada Shannon] You're right that theory bounds can be loose in practice, and I'm not asking you to take this purely on faith either. We're about to look at exactly that — training runs across different depths and different n_h values on state-tracking benchmarks, to see whether the geometry actually predicts what happens. The argument doesn't replace the experiment, it just tells you where to look before you run it, which is already more than most architecture papers hand you.

15
00:07:18,975 --> 00:07:45,850
[Hal Turing] That's a good place to leave the theory hanging for a minute. So here's the question this whole paper is chasing: can a linear RNN gain real, tunable state-tracking expressivity just by taking multiple gradient-descent steps per token instead of one — and does that mechanism actually pay off once you put it in front of real language modeling and long sequences, not just the geometry on paper? Let's go find out whether the numbers back up the intuition.

16
00:07:45,850 --> 00:08:12,100
[Hal Turing] Let's get to the numbers, because this is where the geometry either earns its keep or doesn't. They test DeltaProduct on group word problems for S3, S4, A5, and S5 — Merrill and colleagues' benchmark family for state-tracking. Training runs on sequences of 128 permutations, then they push out to 512 to check extrapolation. So, does more n_h actually buy the generalization you'd hope for?

17
00:08:12,100 --> 00:08:47,200
[Dr. Ada Shannon] Cleanly, yes. S3 needs n_h equals 2 to hold past training length, S5 needs n_h equals 4 — matching the theoretical count of n minus 1 reflections. But here's the surprise: S4 and A5 only need n_h equals 2, despite naive counting predicting 3 and 4. Turns out S4 is isomorphic to the rotation group of a cube, A5 to the rotation group of a dodecahedron — both subgroups of SO(3), which only takes two reflections to generate a rotation. They verified it: learned betas cluster right at 2, pure reflections, and PCA on the keys shows three components explain over 95% of the variance.

18
00:08:47,200 --> 00:09:08,550
[Hal Turing] Oh wait, wait — that's genuinely a chill moment. The model, with zero geometric priors, just discovers the rotation group of a cube on its own, because that's the minimal-description solution actually sitting there in the group theory. That's not luck, that's the network finding the same structure a mathematician would put on a blackboard.

19
00:09:08,550 --> 00:09:27,050
[Dr. Ada Shannon] Exactly — good validation that the theory describes what's actually learned, not just a loose bound. Compare that to fixing n_h at 1 and just adding layers, which is plain DeltaNet: S3 needs 3 layers, S4 needs 6, A5 needs 3. S5 never fits the training length even at 10 layers.

20
00:09:27,050 --> 00:09:49,250
[Hal Turing] I'll push on that, though. Theorem 1 in this paper says four layers at n_h equals 1 CAN solve any group word problem, including S5, in principle. So if it fails at 10 layers empirically, isn't that an optimization problem — bad gradients through ten stacked recurrences — rather than proof depth is architecturally worse than n_h?

21
00:09:49,250 --> 00:10:08,425
[Dr. Ada Shannon] Fair, but look at what that construction demands: a lookup table in the second-to-last layer's MLP, sized two-to-the-m times n-factorial-squared-to-the-m. For S5 that's astronomically wide. It exists on paper — but 'exists' and 'trainable at reasonable width with gradient descent' aren't the same claim.

22
00:10:08,425 --> 00:10:23,275
[Hal Turing] So depth isn't impossible in principle, it's just hiding an exponential tax in that lookup table — a tax n_h pays for directly, in the transition matrix itself, instead of laundering it through an enormous MLP.

23
00:10:23,275 --> 00:10:46,250
[Dr. Ada Shannon] That's the trade-off I'll own — depth gets you there on paper, n_h gets you there in practice. Anyway, the paper doesn't stop at synthetic permutations. They train real language models — DeltaProduct and Gated DeltaProduct, 200 million up to 1.3 billion parameters, on FineWeb. Length extrapolation past the 4096-token training context gets tested on CodeParrot for code, OpenThoughts-Math, and TriviaQA, plus the standard lm-eval-harness suite.

24
00:10:46,250 --> 00:11:00,900
[Hal Turing] Does the story repeat there too — n_h equals 2 as the practical sweet spot — or does real language modeling behave differently once you're past clean synthetic permutations and into actual token prediction?

25
00:11:00,900 --> 00:11:33,900
[Dr. Ada Shannon] It's consistent, and it's the most interesting result here. Going from n_h 1 to 2 produces a sharp drop in loss on out-of-distribution length across CodeParrot, the math set, and TriviaQA. Going from 2 to 3, the curve nearly flattens. The explanation is effective rank of the hidden state: DeltaNet's state keeps accumulating rank the further past 4096 tokens you go, drifting into a regime it never trained on, and loss blows up. DeltaProduct's heads, especially gated ones, learn to reset hard at a new context and let the state decay — rank stays bounded, no distribution shift to fall into.

26
00:11:33,900 --> 00:11:47,475
[Hal Turing] Does that survive once you actually scale up, though, or is this the kind of gap that looks dramatic at 200 million parameters and quietly closes once you throw real scale at plain DeltaNet?

27
00:11:47,475 --> 00:12:21,025
[Dr. Ada Shannon] It holds, at least up to what they tested. Figure 10 matches parameters by scaling head dimension rather than head count — that scaled more consistently — and DeltaProduct keeps its perplexity and lm-eval-harness edge all the way to about 1.3 billion parameters, the largest scale here. The cost is real: the recurrence scales linearly with n_h, so DeltaProduct 3 does three times the sequential work per token versus DeltaNet. They claw back some of that with an optimized Triton kernel 20% faster than DeltaNet's baseline — a constant-factor win on top of a linear cost, not a fix for it.

28
00:12:21,025 --> 00:12:32,225
[Hal Turing] Before we get to whether any of this generalizes beyond the paper's own experiments, give me the two headline theorems in plain English — skip the notation this time.

29
00:12:32,225 --> 00:13:05,925
[Dr. Ada Shannon] Theorem one: for any n_h, DeltaProduct solves any group word problem — S3 up through arbitrarily large symmetric groups — in a bounded number of layers, at most four, fewer as n_h grows. Theorem two covers Gated DeltaProduct: with a forget gate and enough layers, it recognizes any regular language at all, not just permutation groups — the tier of the Chomsky hierarchy that includes things like balanced parentheses and finite-state pattern matching. The upshot: a model provably outside the weak complexity class transformers and diagonal RNNs are stuck in, where n_h decides how many layers you pay to get there.

30
00:13:05,925 --> 00:13:24,625
[Dr. Ada Shannon] Any regular language a finite-state automaton can define — full stop, given enough layers and the right gate. That's the theoretical ceiling. But a theorem that holds for any n_h and any language is a much bigger claim than what actually got run on a GPU, and I want to flag that gap before we close this out.

31
00:13:24,625 --> 00:13:58,700
[Hal Turing] That's exactly where I wanted to push. Every state-tracking result we walked through — S3, S4, A5, S5 — comes from Merrill, Petty, and Sabharwal's synthetic permutation benchmark, out of the Allen Institute for AI, ICML 2024, trained at 128, tested to 512. Does any of that tell us DeltaProduct would track state through a multi-step arithmetic trace, a code interpreter, or an agent chaining tool calls — or is this confined to that one synthetic family?

32
00:13:58,700 --> 00:14:37,550
[Dr. Ada Shannon] Honestly, no — and the paper doesn't claim it does. It's a clean, controlled probe of expressivity, not a downstream evaluation. Composing permutations of five elements is a toy next to tracking variable bindings across executed Python. Same caution applies to scale: FineWeb only, capping at 800 million to 1.3 billion parameters. We've watched architectural gaps between RNNs and Transformers shrink or flip once real scale gets thrown at them, and this paper simply offers no evidence either way past 1.3 billion parameters. Anyone citing it as settling that question at production scale is overreading it.

33
00:14:37,550 --> 00:15:00,700
[Hal Turing] Wait, hold on — sorry to cut in, but that scaling comparison in Figure 10 bugs me for a separate reason. It's parameter-matched, not compute-matched. DeltaProduct's recurrence cost scales linearly with n_h — that's the paper's own admitted limitation — so a matched parameter count is quietly a bigger training budget. Doesn't that inflate the apparent advantage?

34
00:15:00,700 --> 00:15:32,550
[Dr. Ada Shannon] Fair hit, and it's never addressed with a FLOPs-matched curve. Same blind spot with RWKV-7, which sits in their own Table 1 as the closest theoretical rival, proven to recognize any regular language in four layers — yet there's no empirical head-to-head language modeling or state-tracking run against it anywhere. Worth flagging too that the big lookup-table construction behind Theorem 1's harder cases is never actually built. Every experiment stays in the tame n_h one-through-four regime, so the strongest theoretical claims are decorative next to what's tested.

35
00:15:32,550 --> 00:15:52,875
[Hal Turing] Same skepticism should probably apply to the effective-rank story behind length extrapolation. DeltaNet's rank balloons past training length, DeltaProduct's heads learn to decay it, and it correlates cleanly with the loss curves. That correlation is clean enough — I'm inclined to just accept it as the explanation.

36
00:15:52,875 --> 00:16:10,450
[Dr. Ada Shannon] No, no — I actually disagree with you there, Hal. A tidy correlation isn't a causal test, and the paper never runs one. Their own phrase is 'we attribute' — that's a hedge, not a proof. A real test would intervene: clamp DeltaNet's rank down artificially and see if the curves move with it.

37
00:16:10,450 --> 00:16:20,450
[Hal Turing] Isn't that a fully general objection to basically all interpretability work, though? Nobody's clamping attention heads and calling it causal either.

38
00:16:20,450 --> 00:16:34,075
[Dr. Ada Shannon] Sure, and I'd hold those papers to the same bar — I'm not singling this one out. My point is narrower: don't let a pretty picture get promoted to an explanation. Call it a well-supported hypothesis and leave it there.

39
00:16:34,075 --> 00:17:15,425
[Hal Turing] Fair. So where does this leave DeltaProduct in the landscape? The pitch, next to RWKV-7 from Bo Peng and the RWKV team, 2025, is a stability guarantee — spectral norm always at most one — against RWKV-7 trading that away for strictly higher expressivity per layer. There's a blind spot too: a growing line of work treats linear-RNN states as compressible for cheap inference, low-rank caches, quantized state, and DeltaProduct deliberately raises the rank of that state. Nobody's checked whether a rank-3 or rank-4 state still compresses the way DeltaNet's does.

40
00:17:15,425 --> 00:17:49,925
[Dr. Ada Shannon] The authors are candid about what's next: an adaptive version picking n_h per token, in the spirit of Graves' 2016 adaptive computation time, so you don't pay the linear cost everywhere. LoRA-style projections to shrink the extra parameters, following RWKV-7's approach, and combining this with fixed-point RNNs — Movahedi, Sarnthein, Muca Cirone, and Orvieto, out of the ELLIS Institute Tübingen, 2025 — which buys expressivity through iteration in depth instead of a denser matrix. Given state-tracking's link to reasoning, that's the application they want to chase next.

41
00:17:49,925 --> 00:18:30,825
[Hal Turing] So here's where I land. The mechanism is elegant — a single dial, tied to a real theorem, that measurably buys expressivity DeltaNet lacks, and the S4/A5 isomorphism was a genuine surprise. But the validation is narrower than the framing suggests: one benchmark family, one training corpus, sub-1.3-billion scale, no compute-matched comparison, no RWKV-7 head-to-head. Good NeurIPS-caliber science on a specific mechanism — just don't read it as a settled verdict on linear RNNs beating Transformers at scale. That's DeltaProduct. Thanks for listening, and we'll see you next time.