1
00:00:01,000 --> 00:00:39,924
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon. Today we're digging into a paper called "Learning to Solve Hard Problems in RL for LLMs by Never Giving Up," by Michael Noukhovitch et al., four authors total, out of Mila and Université de Montréal, the Allen Institute for AI, University of Washington, and Trillium Labs, posted to arXiv on September 11th, 2026. And Ada, the number that grabbed me right out of the abstract: on AIME math, RL training bumped easy problems by 63 points but hard problems by only 3.

2
00:00:39,924 --> 00:01:15,150
[Dr. Ada Shannon] That gap is basically the whole paper in one chart. What makes it worth a full episode is that this isn't one flaky benchmark — they show the same lopsided pattern on a coding benchmark and an agentic coding benchmark too. Three completely different domains, same shape of curve. If RL post-training is supposed to be the thing that pushes a model past what it learned in pretraining, and it turns out it's mostly just polishing the stuff the model could already sort of do, that's a real problem for anyone betting compute on RL to unlock genuinely hard capabilities.

3
00:01:15,150 --> 00:01:52,775
[Hal Turing] Right, and they're precise about where those numbers come from. They took three open-source RL-trained models — Olmo 3.1 RL-Zero on math, evaluated on AIME; DeepCoder on code, evaluated on LiveCodeBench v5; and DeepSWE on agentic coding, evaluated on SWE-Bench Verified — and compared each one back to its own un-RL'd base model, bucketed by difficulty. Easy AIME problems: plus 63 points. Extra-hard LiveCodeBench: plus 5. Agentic tasks that take a human over four hours: basically plus zero. Same story, three times over.

4
00:01:52,775 --> 00:02:29,100
[Dr. Ada Shannon] And they give that story a name — the Matthew Effect in RL for LLMs. Borrowed straight from Robert Merton's 1968 paper in Science, "The Matthew Effect in Science," where he noticed famous scientists get disproportionate credit for work comparable to what unknown researchers produce — the rich get richer, roughly. Merton pulled the name from the Gospel of Matthew, "for whoever has will be given more." Noukhovitch and company are arguing RL training does the same thing to a model's skills: performance improves in proportion to how good the model already was at that task, so competence compounds instead of spreading evenly.

5
00:02:29,100 --> 00:03:07,125
[Hal Turing] Okay, here's my gut reaction, tell me if I'm being naive — isn't this just the obvious consequence of how these models get trained? RL for LLMs mostly runs on GRPO, Group Relative Policy Optimization. You sample a handful of completions per prompt, score them, and the reward is each completion's score minus the group average. If a problem is hard enough that literally none of your samples get it right, the whole group scores zero, everything equals the average, and there's no gradient at all. No signal, no learning. Feels like hard problems just need more dice rolls.

6
00:03:07,125 --> 00:03:24,350
[Dr. Ada Shannon] That's the intuitive read, and it's got a name in the literature — signal loss, from Xiong et al.'s 2025 work — but I don't think it's the real driver here, and this is where the paper actually picks a fight with the obvious explanation. Wait, sorry, let me back up before you run with it—

7
00:03:24,350 --> 00:03:43,500
[Hal Turing] No no, go ahead, that's exactly the kind of pushback I want. Because the standard fix for signal loss is dead simple: just sample more completions per prompt, K. More rolls of the dice, higher odds one lands on a hard problem. That's basic probability, Ada, I don't see how you dodge that.

8
00:03:43,500 --> 00:04:19,850
[Dr. Ada Shannon] It's basic probability applied to the wrong side of the ledger. The paper's counter-framing is signal efficiency: the claim isn't that hard problems are undersampled, it's that GRPO burns compute oversampling problems the model has already solved. Every completion spent reconfirming an easy prompt for the fifth time is a completion not spent on a prompt still teaching the model something. Cranking up K doesn't fix that trade — it just changes which prompts get discarded for a zero gradient, easy or hard. That's a genuinely different diagnosis, and it points to a different fix: don't sample more everywhere, reallocate where you sample.

9
00:04:19,850 --> 00:04:27,700
[Hal Turing] Fair, that's a sharper distinction than I gave it credit for. So what's the fix — is this where Never Give Up comes in?

10
00:04:27,700 --> 00:05:08,150
[Dr. Ada Shannon] Exactly, though quick disambiguation first — Never Give Up is also the name of a completely different 2020 DeepMind paper by Badia et al. on intrinsic-motivation exploration for Atari agents. Noukhovitch's NGU borrows the name as an homage, not the mechanism — nothing to do with count-based exploration bonuses. Their version: sample a small batch of completions for a prompt, and if none succeed, don't give up — go back and sample more, continuing with probability p and abandoning it otherwise so you're not burning infinite compute on something unsolvable. Because modern RL pipelines run asynchronously, a solved prompt just gets replaced in the queue, and compute naturally drifts toward what's still unsolved.

11
00:05:08,150 --> 00:05:26,900
[Hal Turing] So the sampling policy itself becomes difficulty-aware, without anyone hand-labeling difficulty. Neat inversion. Alright, next up: how they actually test whether signal loss or signal efficiency is telling the truth — and it involves a K-sweep that doesn't go the way you'd expect.

12
00:05:26,900 --> 00:06:03,050
[Hal Turing] They set up a controlled testbed on GSM8k — well, GSM8k Platinum, the cleaned and verified version from Vendrow and colleagues — training Qwen2.5-0.5B-Instruct with GRPO. Crucially, they sorted every eval problem into four difficulty tiers based on the initial model's own pass rate: twenty-five percent for easy, ten for medium, five for hard, and zero for extra-hard — meaning three-quarters of those extra-hard problems never got solved once across a thousand-plus samples.

13
00:06:03,050 --> 00:06:38,875
[Dr. Ada Shannon] Right, and that's the setup for the actual K-sweep. They ran GRPO with K equals four, eight, sixteen, and thirty-two completions per prompt, adjusting the number of prompts per batch so the total batch size — N times K — stayed fixed. Every single setting still shows the Matthew Effect, easy problems improve faster than hard ones no matter what. But here's the twist: K equals four, the smallest setting, wins overall, and it's most visible exactly where you'd expect signal loss to hurt most — the hardest problems. That result alone kills signal loss as the primary explanation.

14
00:06:38,875 --> 00:06:57,800
[Hal Turing] Wait — hold on, hold on. That can't be right. If K equals thirty-two gives you thirty-two independent shots at sampling a correct answer on a hard problem, that should mathematically raise your odds of getting at least one right. How does fewer samples win? I think you're misreading the chart, Ada.

15
00:06:57,800 --> 00:07:17,900
[Dr. Ada Shannon] I'm not misreading it, Hal, I promise. You're only counting one side of the coin. Yes, bigger K raises the odds a hard prompt gets at least one correct sample. But it raises the odds of an easy prompt getting at least one wrong sample by exactly the same amount — and under GRPO's group baseline, that's all it takes to keep an already-solved prompt sitting in the training batch.

16
00:07:17,900 --> 00:07:40,175
[Hal Turing] Okay, but that still feels like it should wash out statistically. Sure, you get more noisy easy-prompt inclusions, but you're also landing real, hard-won correct answers on problems that were previously unsolvable. Shouldn't that upside dominate? I'm not fully sold that bookkeeping alone explains a result this stark. That's the part that still nags at me.

17
00:07:40,175 --> 00:08:07,074
[Dr. Ada Shannon] It would wash out if the two effects cost you the same thing, but they don't. They actually tracked this in Figure 3 — training-batch composition over time. Early on, small K has fewer hard prompts in the batch, same as you'd guess, it hasn't found any yet. But by around step two hundred, K equals four has filtered out easy prompts so aggressively that its batches are dominated by hard problems, while K equals thirty-two stays diluted with easy ones it never shakes loose. It's not bookkeeping — it's where your gradient budget actually goes.

18
00:08:07,074 --> 00:08:32,774
[Hal Turing] Okay, that I actually buy — it's compute allocation, not raw sampling odds, and once you see it that way, K equals four almost accidentally stumbled onto the right idea. Which, I'm guessing, is exactly the gap Never Give Up is built to close on purpose instead of by accident. So give me the actual numbers on it — how do they pick that give-up probability, and what happens to all the failed completions along the way?

19
00:08:32,774 --> 00:09:16,600
[Dr. Ada Shannon] Right — same K-then-resample-with-probability-p idea as before, just with a name: p sub NGU. That creates a geometric distribution over total samples, with an expected size of K over one minus p. Run with p equals zero point nine five on GSM8k, and the pass-at-one gains land almost entirely on the extra-hard tier, with batch composition close to pareto-optimal — best of small-K filtering and large-K persistence together. One wrinkle: they keep prior completions around to build one big GRPO group once a prompt finally gets solved, sharpening the advantage on rare correct answers, but completions older than about four steps start hurting performance, so they cap staleness and use a rescaling trick called anchoring the positives to keep old negative signal without discarding it outright.

20
00:09:16,600 --> 00:09:37,200
[Hal Turing] And this isn't just a small-model party trick, presumably — they pushed it further, right? Showing a clean result on a half-billion-parameter model doing grade-school arithmetic is one thing. What happened once they scaled up to real competition math, and to a much harder coding benchmark? I'm curious how far this generalizes.

21
00:09:37,200 --> 00:10:41,975
[Dr. Ada Shannon] On math, they scaled to Qwen3-4B-base on Deepscaler, testing against real contest problems — AIME 2025 and BRUMO. Same pattern: raising K trades off easy against hard, but NGU trades it off better, and it beats a curriculum baseline that sets K per-prompt by initial difficulty — that baseline matches NGU on hard problems but tanks on easy ones, because difficulty estimates go stale as training proceeds. Then on Manufactoria, Sun and colleagues' 2025 coding benchmark, training Qwen3-4B-Instruct with per-test rewards, they hit something stranger: no Matthew Effect at first, because the model hasn't adapted to this unfamiliar harness yet — re-measure difficulty after about a hundred steps and it reappears clearly. Standard GRPO plateaus around eighty percent of tests solved but almost never clears every test on a problem; NGU keeps grinding on the hard tests where GRPO stalls and eventually learns to pass full problems. And there's a nice coda — recovering cleanly from three thousand steps of a 'wrong' objective is notably not what Nikishin and colleagues found for plasticity loss in from-scratch deep RL, out of Mila, back in 2022.

22
00:10:41,975 --> 00:11:15,925
[Hal Turing] Okay, something's been nagging at me since we walked through Figure 1 — the whole motivating case for the Matthew Effect. That evidence came from Olmo 3 7B, DeepSeek-Qwen-14B, and Qwen3-32B. But every NGU result we just went through tops out at Qwen3-4B for Deepscaler and Manufactoria, and half a billion parameters for the core GSM8k mechanism study. So the thing that convinced us the problem exists was measured on much bigger models than the thing that claims to fix it.

23
00:11:15,925 --> 00:12:00,225
[Dr. Ada Shannon] Right, and the paper doesn't paper over that gap, but it doesn't close it either. The honest answer is we don't know if compute reallocation holds the same value at 30B-plus, where production RL post-training actually lives. A bigger base model might have a sharper, more bimodal pass@1 distribution — more prompts near-certain either way, fewer sitting in that useful medium band NGU thrives on reallocating from. That could shift the easy-to-hard sample ratio enough to change the whole cost-benefit case. And it compounds with another loose end — p-NGU, the give-up probability, is basically hand-set per experiment: 0.95 on GSM8k and Manufactoria, swept over half a dozen values on Deepscaler. There's no principled rule for picking it in a new domain, and Appendix C.1 shows p equals one can actually stall training by endlessly resampling an unsolvable prompt.

24
00:12:00,225 --> 00:12:23,725
[Hal Turing] That connects to something else that bugged me — the harness-aware redefinition in Section 6. They look for the Matthew Effect at initialization on Manufactoria, don't find it, then re-measure difficulty after a hundred adaptation steps and — surprise — there it is. That's a little too convenient. How do we know they didn't just keep changing the measurement until it matched what they already believed?

25
00:12:23,725 --> 00:12:57,325
[Dr. Ada Shannon] No, hold on, hold on — that's not fair to what's actually in Figure 8. Look at the mechanism they give for why initialization is a bad difficulty label in the first place: Manufactoria's prompt format is genuinely novel, the model hasn't adapted to the harness yet, so early pass rates are measuring prompt-following, not task competence. That's a testable claim independent of whether the Matthew Effect shows up afterward — they're not just p-hacking a knob until a chart looks right, they're arguing the pre-adaptation measurement is confounded on structural grounds.

26
00:12:57,325 --> 00:13:22,400
[Hal Turing] I hear that, and it's a real argument, not just an ad hoc excuse — I'll give you that. But it's still a post-hoc justification chosen after seeing which framing produced the effect, and they don't show it on a task where the harness isn't novel, so we can't cross-check it. I'll let it stand as plausible rather than proven. Different question, though — how does NGU actually differ from Xiong et al.'s Reinforce-Ada, since it sounds like a similar fix?

27
00:13:22,400 --> 00:14:20,800
[Dr. Ada Shannon] Same target, different diagnosis. Reinforce-Ada frames it as signal loss and stays synchronous, resampling within a batch. NGU needs asynchronous RL — Noukhovitch et al., 2024 — because the whole trick is refilling a queue slot the instant a prompt resolves, which a synchronous method structurally can't do. Practically, this matters because async infrastructure is already where the industry's headed — Cursor's Composer 2, GLM-5 — so NGU isn't asking teams to adopt new plumbing, just to spend the plumbing they're building anyway more intelligently. And interestingly, Courville's got a parallel thread here too — he's also on Stable Deep RL via Isotropic Gaussian Representations, so this compute-allocation angle isn't a one-off for him. The authors are upfront that everything here is contextual-bandit RL, single verifiable answer, not multi-step — extending this to agentic tool-use trajectories, where 'correct' is a sparse outcome over many turns, is flagged explicitly as future work.

28
00:14:20,800 --> 00:14:46,475
[Hal Turing] Which is the honest note to end on, I think — this paper's real contribution isn't NGU the algorithm, it's the reframe: scalar pass-or-fail rewards are hiding a compute allocation problem that 'just add more samples' can't solve, and might even worsen. Whether that holds at 32B or in agentic settings is genuinely open, and they say so. That's it for this one — thanks for listening, and we'll catch you next time.