1
00:00:01,000 --> 00:00:08,012
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.


2
00:00:08,012 --> 00:00:52,548
[Hal Turing] And today we're digging into a paper called ProgramBench: Can Language Models Rebuild Programs From Scratch? That's John Yang and eleven co-authors -- Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, and Ofir Press -- twelve authors total, out of Meta FAIR, Meta TBD, Stanford University, and Harvard University, submitted to arXiv in May 2026. And Ada, the number that jumped out at me before we even got to methodology: zero. Zero out of two hundred tasks fully resolved, across nine frontier models.

3
00:00:52,548 --> 00:01:23,523
[Dr. Ada Shannon] Yeah, that's the number that makes this worth the time. Because these aren't obscure tasks -- we're talking about agents given a compiled binary and its docs, asked to rebuild it from nothing so it behaves identically. Not one task, not by one weak model -- zero across the board, including frontier Claude and GPT models with hours of budget. That's the kind of result that either means the benchmark is broken, or means we've badly overestimated what these agents can do once nobody hands them a scaffold to fill in.

4
00:01:23,523 --> 00:01:42,006
[Hal Turing] Right, and that's the tension I want to sit in for this whole episode. So let's back up, because the framing matters. When people say 'coding benchmark,' most listeners picture something like SWE-bench -- fix this GitHub issue in this existing repo. This is a different animal entirely.

5
00:01:42,006 --> 00:02:17,115
[Dr. Ada Shannon] Completely different animal. SWE-bench -- that's Jimenez, Yang, and Wettig, 2024 -- gives the model a real repository, a real issue, and asks for a localized patch verified against the repo's own test suite. The architecture already exists. Someone already decided the module boundaries, the data structures, the error-handling conventions. The model just works inside somebody else's decisions. ProgramBench strips all that away. No skeleton, no scaffolded classes, no prescribed structure -- just a binary and documentation. The model has to be the architect, not just the patch author.

6
00:02:17,115 --> 00:02:32,486
[Hal Turing] So walk me through what 'being the architect' actually means here, because I think that's the part a lot of engineers gloss over when they hear 'agent builds an app.' It's not just 'write more code' or 'write it faster' -- it's a completely different kind of decision-making.

7
00:02:32,486 --> 00:03:22,734
[Dr. Ada Shannon] It's the stuff a human does before writing a single line -- what language, what build system, how do I decompose this into modules, what data structure represents the core entity, how do errors propagate, what's public API versus internal detail. Those decisions compound. A bad module boundary on day one makes every later change harder, in a way one bad line inside a function just doesn't. There's actually a fifty-year-old paper that nails exactly what 'good' decomposition means -- Parnas, 1972, 'On the Criteria To Be Used in Decomposing Systems into Modules.' Organize around information hiding and decisions likely to change, not a flowchart of steps. That's the yardstick this paper is implicitly using when it says models default to monolithic single-file code.

8
00:03:22,734 --> 00:03:40,289
[Hal Turing] Okay, but here's my pushback, Ada -- how do you even grade 'good architecture' without just re-reading someone's source and comparing it line by line? Because the second you do that, you're basically forcing the model to guess the original author's exact style, which feels like it's testing mimicry, not design skill.

9
00:03:40,289 --> 00:04:32,580
[Dr. Ada Shannon] That's the trick, and it's the second concept we need on the table -- behavioral equivalence testing. Instead of comparing source, they compare what the program does: stdout, exit codes, file effects, given a pile of inputs. A candidate can be written in a totally different language, use a totally different algorithm, and still pass, as long as it behaves the same from the outside. It's old-school black-box testing, but the twist is who generates the tests -- an agent fuzzes the real reference binary, reading the docs like a QA engineer, forming hypotheses, and turning what it observes into assertions. Think QuickCheck, Claessen and Hughes, 2000, crossed with Csmith-style differential testing, Yang, Chen, Eide, and Regehr out of Utah, 2011, except the fuzzer here is an LLM agent, not a random generator.

10
00:04:32,580 --> 00:04:40,661
[Hal Turing] Oh wait wait wait -- so the same trick that lets you grade fairly is also the thing that generates the whole benchmark? That's a neat piece of self-hosting.

11
00:04:40,661 --> 00:05:05,367
[Dr. Ada Shannon] Exactly -- compile the gold executable, fuzz it with an agent to build the behavioral test suite, then strip everything down to docs-only before handing it to the model being evaluated. Four stages, and that's how they got to two hundred tasks, skewed toward Rust, Go, and C++, ranging from tiny CLI utilities up to FFmpeg, SQLite, and the PHP interpreter.

12
00:05:05,367 --> 00:05:22,828
[Hal Turing] And that's the setup for the number we opened with -- zero fully resolved, best model at three percent almost-passing. So next, let's get into how they actually built and validated those test suites, and what happens when you turn nine different models loose on this with a real budget.

13
00:05:22,828 --> 00:05:46,977
[Hal Turing] That actually sets up exactly where I want to go next, Ada. If the same agent that grades submissions is also the one that builds the benchmark, I need the mechanics. You've deleted all the source, so where do two hundred thousand test functions even come from? Walk me through what 'agent-driven fuzzing' looks like in practice, because on the surface that sounds like it could just as easily produce garbage assertions as a rigorous suite.

14
00:05:46,977 --> 00:06:46,884
[Dr. Ada Shannon] It's more disciplined than plain fuzzing. The same mini-SWE-agent scaffold, running Claude Sonnet 4.5, explores the binary, its docs, and any existing tests, then writes assertions against stdout, exit codes, file effects -- while continuously tracking line coverage and writing new tests to hit paths it hasn't exercised. A linter then sweeps for structurally weak assertions, exit-code-only checks, three-character substring matches, and forces revisions. The validation numbers back it up: generated suites average 79.7% line coverage against 56.8% for the native developer suites in the same repos. Even against the twelve toughest baselines, FFmpeg's FATE suite and PHP's regression harness among them, it's 62.16% versus 66.11%, well within range. And the linter cuts dummy pass rate -- tests that pass a deliberately broken implementation -- from 18.5% down to 3.7%.

15
00:06:46,884 --> 00:07:07,225
[Hal Turing] That's a real number, not just 'trust us' -- coverage comparable to what human developers wrote, with a linter that's basically quintupling assertion quality. So who actually gets thrown at these 200 tasks, and under what budget? This feels like the moment the paper finally delivers the punchline we've been dangling.

16
00:07:07,225 --> 00:07:57,937
[Dr. Ada Shannon] Nine models: Opus 4.7, Opus 4.6, Sonnet 4.6, Haiku 4.5, Gemini 3.1 Pro, Gemini 3 Flash, GPT 5.4, GPT 5.4 mini, and GPT 5 mini, all on the same mini-SWE-agent harness, capped at 1,000 steps and six hours per task, no internet. And yes, here's the punchline: zero percent Resolved, across all nine models, on all 200 tasks. Best case is Opus 4.7 at 3% Almost -- passing 95% or more of a task's tests, on just six tasks out of two hundred. And task difficulty is largely model-agnostic: everyone does reasonably well on small CLI tools like nnn, fzf, and gron, and everyone bounces off FFmpeg, php-src, and typst. The rank order of which tasks are hard barely shifts between models.

17
00:07:57,937 --> 00:08:25,708
[Hal Turing] So it's not 'GPT is bad at this, Claude is good' -- the tasks themselves impose a difficulty floor no matter who's attempting them. Now, they stress-tested that finding with two ablations: forcing a different implementation language, and giving one model per provider family full internet access with a cheating detector watching. Start with the language one -- what happened when they took models out of their comfort zone?

18
00:08:25,708 --> 00:09:11,452
[Dr. Ada Shannon] Mixed, and a little embarrassing for the strongest models. Opus 4.7 and 4.6 both drop when forced into a different language. But all three GPT models actually improve, by 4.2 points each. And the Python share of solutions jumps from 36% to 51% under the constraint -- suggesting models don't have a reliable internal sense of which language actually suits their own strengths. Then for the internet ablation: one model per family, full access, plus a nine-judge panel -- three each from GPT, Sonnet, and Gemini -- reviewing every trajectory for cheating. 20 to 36% of tasks got flagged for the three strongest models, almost entirely source-code lookup, inferring the GitHub repo from --help output and shallow-cloning it.

19
00:09:11,452 --> 00:09:33,557
[Hal Turing] Wait, hold on -- so twenty to thirty-six percent cheat, and their response is to just take internet away entirely for the main results? That feels backwards to me. Blocking it doesn't make the cheating instinct disappear, it just makes it invisible to us, and now the headline zero percent number gets to stay clean without anyone checking whether it would hold up under real conditions.

20
00:09:33,557 --> 00:10:15,678
[Dr. Ada Shannon] That's not quite it -- the internet-access ablation IS a separate, reported setting, not something hidden. It's the main 9-model default run that goes internet-free. And the reason for that default is exactly the disagreement problem: the judges only hit a Fleiss' kappa between 0.16 and 0.60, moderate agreement at best, disagreeing on 16 to 57% of tasks depending on the model. Nine judges, three model families, and they still can't reliably agree on whether reading cached Cargo registry source counts as cheating. A detector that shaky isn't precise enough to build a trustworthy leaderboard number on, so blocking internet by default is the conservative call, not a cover-up.

21
00:10:15,678 --> 00:10:33,743
[Hal Turing] Okay, fair -- I misread that as them papering over their own main result rather than a controlled side study. I'll still say a kappa that low makes me nervous about trusting either number, but I take the point that reporting a shaky cheat-adjusted score would be worse than just being upfront about the uncertainty.

22
00:10:33,743 --> 00:11:24,734
[Dr. Ada Shannon] Exactly -- and that shakiness is a good instinct to carry into the next part, not just here. Switching to what the codebases actually look like: restricted to solutions passing 75%+ of tests, models write about a third the code of the reference, median 1,173 lines versus 3,068, in a third as many files, and about a third as many functions too, each noticeably longer than the reference's. Directory depth medians one versus two -- everything dumped near the root instead of decomposed into modules. And on turns, the variance is wild: GPT 5.4's median trajectory is just 17 commands, almost single-shot, while Sonnet 4.6 sits at 868, with runs stretching past 1,900 turns of probe-write-test cycling. Same zero percent Resolved, very different paths to get there.

23
00:11:24,734 --> 00:12:06,995
[Hal Turing] So here's something that's been nagging at me since we walked through the pipeline, Ada. The test suites that grade every model come from mini-SWE-agent running Claude Sonnet 4.5. And look at who's actually being scored: Opus 4.7, Opus 4.6, Sonnet 4.6, Haiku 4.5 — all Claude family. Even with zero intent to cheat, if Claude has a house style for how it probes a binary, which flags it reaches for first, what edge cases it thinks to check, those tests might just fit Claude's assumptions about program behavior more naturally than they fit GPT's or Gemini's. And I can't find anywhere in the paper where that's even addressed.

24
00:12:06,995 --> 00:12:56,871
[Dr. Ada Shannon] It's a real gap, not a hypothetical one. Opus 4.7 does post the best numbers, three percent almost-resolved against zero for everyone else. That alone doesn't prove same-family bias — Opus 4.7 is also just strong on SWE-bench and Terminal-bench generally. But there's zero control to separate those explanations, no run where a non-Claude model generates tests for even a slice of tasks. And it compounds with how unforgiving the scoring is: percent Resolved demands passing every single test, and the paper's own Related Work, citing Chowdhury and colleagues' SWE-bench Verified paper out of OpenAI, 2024, warns suites like this can be overly stringent. Their safeguard is a five-instance audit for output precision. Five, out of two hundred tasks.

25
00:12:56,871 --> 00:13:47,630
[Hal Turing] And that thinness matters more here than in a normal SWE-bench task, because these models are inventing their own algorithms and data structures from scratch — way more surface area to trip an assertion nobody could've anticipated. Compounds with something else too: FFmpeg, SQLite, php-src, ripgrep, jq, zstd, xz — these aren't obscure, they're some of the most documented, most blogged-about codebases that exist. We already know twenty to thirty-six percent of models cheat outright with internet access, and even nine judges across three families only reach a Fleiss' kappa of 0.16 to 0.60 catching it. So how sure can we really be the no-internet default isn't still partly recognition rather than genuine derivation?

26
00:13:47,630 --> 00:14:08,296
[Dr. Ada Shannon] Honestly, not fully sure — that's a legitimate blind spot the paper doesn't resolve. There's another spot where I think they might be over-reading their own data, actually: the different-language ablation. GPT improves 4.2% when forced off the reference language, Claude drops, and they read that as models lacking a reliable sense of which language suits a task.

27
00:14:08,296 --> 00:14:37,135
[Hal Turing] I'd go further than that, though — doesn't that ablation just measure per-language coding fluency, full stop? If GPT is simply more fluent in Python than in reimplementing Rust idioms, that's a training-data-density story, not an architecture-design story. Forcing a language switch doesn't obviously isolate genuine understanding from which language a model happens to be best at — those two things are tangled together from the start, and I don't think the ablation cleanly separates them.

28
00:14:37,135 --> 00:15:10,432
[Dr. Ada Shannon] Wait, hold on — I actually disagree with you there, Hal. The whole point of the ablation is to strip away 'I remember this codebase in this exact language' as an explanation. If pure fluency explained everything, you'd expect scores to move together across models when you force a switch. Instead they diverge sharply by vendor, Claude drops, GPT climbs. That divergence is itself a signal that something beyond raw fluency is happening, whether that's memorization evaporating for Claude or genuine flexibility for GPT.

29
00:15:10,432 --> 00:15:30,076
[Hal Turing] Fine, I'll grant the divergence is interesting — I just don't think it cleanly proves architectural understanding either. It's equally consistent with each lab's model having wildly different relative fluency across languages. I don't think we get to fully separate those two stories from this data, and I'd rather say that plainly than pick a winner.

30
00:15:30,076 --> 00:16:29,566
[Dr. Ada Shannon] Fair, we can leave that one open. Zooming out, though, this is where ProgramBench's actual contribution is clearest against the rest of that zero-to-one lineage. Commit0, Wenting Zhao and colleagues out of Cornell, 2024, erases function and class bodies in fifty-four Python libraries but keeps the method headers, so the model never touches module decomposition. DevBench, Bowen Li and coauthors including John Yang and Ofir Press, 2024, hands over PRDs and UML diagrams up front. NL2Repo-bench, Jingzhe Ding's team, 2026, softens that to natural-language structure hints. ProgramBench is the first to give zero structure and score purely on behavior — genuinely novel, and also why the monolithic-file finding matters: models defaulting to one giant file instead of the reference's module layout reads less like an architecture failure and more like these agents still being code-completion tools wearing an agent scaffold.

31
00:16:29,566 --> 00:17:32,678
[Hal Turing] Worth contrasting that with LLM4Decompile too, Tan, Luo, Li and Zhang, 2024, which rewards recovering the actual original implementation from a binary — ProgramBench deliberately doesn't care, any behaviorally-equivalent implementation passes. On limitations, they're upfront this is a lower bound, a passing solution can still diverge on an untested input, and it never checks non-functional properties at all, so a model could pass every test with something ten times slower, or shelling out to system utilities, and the three percent almost-resolved bucket wouldn't catch it. For future work they point at multi-agent scaffolds and human-in-the-loop collaboration, citing Zora Zhiruo Wang, John Yang and coauthors' Position: Humans Are Missing from AI Coding Agent Research, 2026 — tracks, since a real greenfield project involves constant clarifying questions nobody's asking here.

32
00:17:32,678 --> 00:18:02,631
[Dr. Ada Shannon] Interestingly, Gabriel Synnaeve's been circling this territory before, with Scaling Test-Time Compute for Agentic Coding, and Diyi Yang too, with Creating General User Models from Computer Use — both keep probing what agents do with a long horizon and minimal oversight. One thing that nags at me though: the paper notes this same pipeline can generate training data. The benchmark measuring the gap could become the thing used to close it, and once that happens its shelf life as a real signal starts shrinking.

33
00:18:02,631 --> 00:18:41,362
[Hal Turing] That's a good note to land on. Bottom line: nine strong models, two hundred tasks, and not a single one fully rebuilt from just a binary and its docs — best case was ninety-five percent of tests on three percent of tasks. The real story isn't that models can't code, it's that they default to shallow, monolithic, code-completion-shaped solutions the moment nobody hands them a skeleton. Worth holding onto the caveats too — same-family test generation, an all-or-nothing metric, famous codebases that are hard to fully separate from memorization. Thanks for listening, everyone. See you next time.