1
00:00:01,000 --> 00:00:07,455
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,455 --> 00:00:47,672
[Hal Turing] Today's paper is "Joules-to-GDP: How Much Economic Value Do AI Systems Produce per Unit of Energy?" The first authors are Amirreza Zeinali, Jon Saad-Falcon and Avanika Narayan, with equal contribution. It comes from Stanford, with UC Berkeley and UT Austin, and senior authors include John Hennessy, Christopher Ré and Azalia Mirhoseini. It's on arXiv, version 2, from November 2026. The question is how much economic value an AI system produces for each joule it burns doing real professional work.

3
00:00:47,672 --> 00:01:06,712
[Dr. Ada Shannon] The claim is that ranking models by capability alone can mislead, because the most capable model isn't necessarily the best use of energy. The same team built Intelligence per Watt, and the compute credits came partly from Modal, Google Cloud, IBM and others. First we need the vocabulary, then the results.

4
00:01:06,712 --> 00:01:11,263
[Hal Turing] Start with why a joule. We already have dollars per token and seconds per task.

5
00:01:11,263 --> 00:01:31,557
[Dr. Ada Shannon] The authors give three reasons. A joule is a stable physical unit, while API prices change every few months. End-to-end energy folds in hardware, tool calls, waiting and retries in one number. And they say energy can rank systems differently from cost or time. That last one is a claim to test, not a given. We'll see whether the data supports it.

6
00:01:31,557 --> 00:01:32,951
[Hal Turing] So what's the prior art?

7
00:01:32,951 --> 00:02:00,814
[Dr. Ada Shannon] Two literatures that mostly ignore each other. On the value side there's GDPval, from Patwardhan and colleagues at OpenAI in 2025. It's a benchmark of real professional tasks, each with source files, a rubric and a deliverable made by an expert. This paper uses the 220 public gold tasks: 44 occupations, five tasks each. It's one-shot, so there's no iterative workflow, and that matters for what it can say about real jobs.

8
00:02:00,814 --> 00:02:21,805
[Dr. Ada Shannon] The closest prior art is Cost-of-Pass, by Erol, El, Suzgun, Yuksekgonul and Zou, ICLR 2026. It asks for the expected dollar cost of getting one correct answer, counting failures and retries, and compares that to a human expert's cost. Joules-to-GDP swaps the dollar denominator for energy.

9
00:02:21,805 --> 00:02:23,106
[Hal Turing] And the energy side?

10
00:02:23,106 --> 00:02:40,521
[Dr. Ada Shannon] ML.ENERGY from Jae-Won Chung and colleagues at Michigan, MLPerf Power, and Intelligence per Watt from Saad-Falcon and colleagues in 2025. They measure joules carefully, but not what the joules bought. The gap this paper fills is value per unit of energy, for real work.

11
00:02:40,521 --> 00:02:44,143
[Hal Turing] Okay, define the metrics. How does a model's output become dollars?

12
00:02:44,143 --> 00:03:06,248
[Dr. Ada Shannon] Output dollar value is the expert's recorded completion time multiplied by the occupation's median hourly wage, from the Bureau of Labor Statistics. The AI earns that amount only if its deliverable wins or ties against the expert's in a blinded pairwise comparison. If it loses, it earns zero. So it's a wage-times-time proxy, not measured GDP.

13
00:03:06,248 --> 00:03:07,409
[Hal Turing] Who does the comparing?

14
00:03:07,409 --> 00:03:19,995
[Dr. Ada Shannon] An LLM judge, Gemini 3.1 Flash-Lite. That's a small, cheap model, where GDPval itself used expert graders. Keep that in mind, because every dollar in the numerator passes through it.

15
00:03:19,995 --> 00:03:21,341
[Hal Turing] And the headline metrics?

16
00:03:21,341 --> 00:03:46,512
[Dr. Ada Shannon] Net Value per Joule, NVJ, is the credited output value minus the AI's execution cost, divided by energy, in dollars per megajoule. Every attempt pays cost and energy, including losses and timeouts. Time Saving per Joule, TSJ, is the expert's time minus the AI's time on wins and ties, divided by energy, in hours per megajoule. Higher is better for both.

17
00:03:46,512 --> 00:03:48,323
[Hal Turing] What's a megajoule in human terms?

18
00:03:48,323 --> 00:04:07,456
[Dr. Ada Shannon] The paper's own equivalence is about 23 hours of active laptop use at 30 watts. Energy is metered with the Intelligence per Watt tooling: GPU, CPU and DRAM over the whole batch window, covering inference, tool use and idle waiting. Facility overhead isn't metered.

19
00:04:07,456 --> 00:04:09,732
[Hal Turing] The human side has two baselines too, right?

20
00:04:09,732 --> 00:04:23,896
[Dr. Ada Shannon] Yes. Expert-only is the primary one. The harder baseline is expert-with-AI, a deliverable made by an expert using Claude through a chat interface, with its own recorded time. A machine that beats a lone expert might not beat an expert with an assistant.

21
00:04:23,896 --> 00:04:26,868
[Hal Turing] One more piece of jargon I saw is tensor parallelism.

22
00:04:26,868 --> 00:04:50,831
[Dr. Ada Shannon] Tensor parallelism across eight GPUs, TP8, splits one model across eight devices. The consequence is that all eight draw power for the entire run. Decode is memory-bandwidth-bound and agent tool calls are sequential, so a small model on eight GPUs can pay for a lot of hardware that sits partly idle. Hold onto that. It matters when we get to the hardware.

23
00:04:50,831 --> 00:04:52,689
[Hal Turing] So what are the questions the paper asks?

24
00:04:52,689 --> 00:05:01,651
[Dr. Ada Shannon] Three. How does return per joule vary by occupation? How do models trade value against energy? And how do accelerator type and GPU count matter?

25
00:05:01,651 --> 00:05:06,063
[Hal Turing] Then I'm already wondering whether the answers come from the models or from the machines they ran on.

26
00:05:06,063 --> 00:05:08,664
[Dr. Ada Shannon] That's the right thing to wonder. We'll get there.

27
00:05:08,664 --> 00:05:12,147
[Hal Turing] Before any results, I need the setup. What exactly did they run?

28
00:05:12,147 --> 00:05:42,333
[Dr. Ada Shannon] The 220 public GDPval gold tasks, 44 occupations, five tasks each. Each model runs inside the OpenHands agent harness, served with vLLM, 20 requests at a time, with a 32,768-token output cap. Five models. Four of them ran on eight H100s. GLM 5.2 ran on eight H200s because its FP8 build didn't fit on eight H100s. The paper says so in a footnote. It matters a lot for the model comparison, and we'll test it properly later.

29
00:05:42,333 --> 00:05:46,791
[Hal Turing] Energy is metered for the whole batch. How does a single task get its share?

30
00:05:46,791 --> 00:06:05,321
[Dr. Ada Shannon] The meter gives one total for the batch window. The authors split it by who is running at each moment. Whoever is active shares the power draw of that slice of time. That matters because of stragglers. One long task keeps the node powered while the other requests have finished, and the long task absorbs that idle-time energy.

31
00:06:05,321 --> 00:06:06,435
[Hal Turing] Is the skew big?

32
00:06:06,435 --> 00:06:25,150
[Dr. Ada Shannon] Figure 8 shows it. Half the requests finish after about 52% of the batch's energy. Ninety percent finish after about 84%. So the last tenth of requests account for roughly a sixth of the energy. One caveat: those intermediate points use a constant-power approximation.

33
00:06:25,150 --> 00:06:28,076
[Hal Turing] And aggregation? Averaging ratios is where papers go wrong.

34
00:06:28,076 --> 00:06:42,798
[Dr. Ada Shannon] They avoid that. Occupation weights are sector GDP share times the occupation's share of employment within its sector. They average value, cost and energy across tasks first, then form the ratio. So the headline is one noisy numerator over one noisy denominator.

35
00:06:42,798 --> 00:06:44,284
[Hal Turing] Start with occupations, then.

36
00:06:44,284 --> 00:07:11,405
[Dr. Ada Shannon] One example is enough. With GLM 5.2, Architecture and Engineering yields about 10% more net value per task than Computer and Math. But Computer and Math yields about 27% more per joule, 523.8 against 413.9. The ranking flips because the denominator changed. The tails are also wide. Production's interval runs from negative 4.24 to 479.79, from five tasks.

37
00:07:11,405 --> 00:07:13,634
[Hal Turing] So I shouldn't memorize any occupation ranking.

38
00:07:13,634 --> 00:07:17,210
[Dr. Ada Shannon] Right. Read it as a demonstration that value and joules can disagree.

39
00:07:17,210 --> 00:07:19,439
[Hal Turing] Models next. Does the most capable one win?

40
00:07:19,439 --> 00:07:51,622
[Dr. Ada Shannon] On value per task, yes. GLM 5.2 earns about $74 per task. The abstract says $72.27, but the body and tables say $74.19, so I'm using that. Gemma 4 31B earns a bit over half of that, $41. But it uses under a quarter of the energy. GLM takes 0.43 megajoules per task, about four hours of laptop use. Gemma takes 0.097, under an hour. Net value per joule is roughly 2.5 times higher for Gemma.

41
00:07:51,622 --> 00:07:53,247
[Hal Turing] How solid is 2.5 times?

42
00:07:53,247 --> 00:08:06,482
[Dr. Ada Shannon] Gemma's interval lower bound just clears GLM's upper bound. The other three models' intervals overlap GLM's. And those intervals only reflect which tasks were drawn. There's one run per task, so no seed variance.

43
00:08:06,482 --> 00:08:12,241
[Hal Turing] The second comparison is the one I care about. GPT-OSS 120B against GLM.

44
00:08:12,241 --> 00:08:44,842
[Dr. Ada Shannon] They look the same on the bill and the clock. GLM costs $1.92 a task and takes 22.7 minutes. GPT-OSS 120B costs about 3% more and runs about 2% faster. But it uses 68% less energy, 0.137 megajoules against 0.430. A cost-of-pass style accounting would call them a tie. On value per AI dollar and per AI hour, GLM comes out ahead. On value per joule, GPT-OSS comes out about 1.6 times ahead. The ranking flips.

45
00:08:44,842 --> 00:08:48,139
[Hal Turing] Is that energy telling us something, or is it a pricing artifact?

46
00:08:48,139 --> 00:09:06,808
[Dr. Ada Shannon] The paper can't separate them. The dollar cost comes from node-hour rates, AWS p5 versus p5en, and the two models ran on different hardware. A flat node rate lumps GPU, host and idle time together. The meter sees what the rate hides. So some of the flip is probably real, but I wouldn't say how much.

47
00:09:06,808 --> 00:09:10,291
[Hal Turing] Then accelerators. This sounds like it's headed toward that footnote.

48
00:09:10,291 --> 00:09:40,895
[Dr. Ada Shannon] It is. Take Qwen3.6-27B. On eight H100s it earns about $68 per task and uses 0.516 megajoules. On one H100 it earns about $65 and uses 0.159. That's 3.25 times the energy for nearly the same value. Completion is only 22% faster on eight, because all eight GPUs draw power the whole run and the tool calls are sequential. Net value per joule goes from 402 on one GPU to 123 on eight.

49
00:09:40,895 --> 00:09:41,963
[Hal Turing] Other accelerators?

50
00:09:41,963 --> 00:09:59,935
[Dr. Ada Shannon] On a single GPU, B200 leads at 535, but its interval overlaps H100 and H200. MI350X sits far below at 124. The paper doesn't explain that gap. It says architecture, bandwidth and power help explain the spread.

51
00:09:59,935 --> 00:10:09,037
[Hal Turing] Hold on. Qwen on one H100 scores 402. Gemma's 417 was on eight H100s. And the same Qwen on eight is 123.

52
00:10:09,037 --> 00:10:26,034
[Dr. Ada Shannon] Yes. Gemma and Qwen are both dense models of 27 to 31 billion parameters. On the same eight-GPU node, Gemma is about 3.4 times better than Qwen. Moving Qwen to one GPU gains about 3.3 times. So the hardware effect is about the size of the model gap.

53
00:10:26,034 --> 00:10:27,753
[Hal Turing] The authors don't say that, do they?

54
00:10:27,753 --> 00:10:39,316
[Dr. Ada Shannon] No. That's our inference from their numbers. They never ran Gemma at a minimal GPU count. So the node effect is large enough to explain much of the Gemma-versus-GLM headline, but nobody knows the exact share.

55
00:10:39,316 --> 00:10:42,056
[Hal Turing] So what would it take to settle the hardware question?

56
00:10:42,056 --> 00:11:12,567
[Dr. Ada Shannon] Running Gemma and the GPT-OSS models at their minimal GPU counts, which the paper doesn't do. GLM's H200 node also has a different power profile, and it may not need eight GPUs at lower precision. My inference, not a result: minimal GPU counts would probably widen the gap for models that fit on one card, and they would change every ratio in the table. So the qualitative claim that small models return more per joule survives. The 2.5 times figure belongs to this node configuration.

57
00:11:12,567 --> 00:11:15,725
[Hal Turing] Next worry. The headline is one noisy number divided by another.

58
00:11:15,725 --> 00:11:38,666
[Dr. Ada Shannon] Yes. One run per task, five tasks per occupation, and the intervals only resample tasks. They don't capture run-to-run variation. The paper's own check is the most telling figure here. On repeated GLM outputs, the win-or-lose decision matched on 77.7% of tasks, 171 of 220. The same model on the same task flips about one time in four.

59
00:11:38,666 --> 00:11:44,703
[Hal Turing] So the occupation flip, with Computer and Math beating Architecture and Engineering per joule by 27%...

60
00:11:44,703 --> 00:12:19,023
[Dr. Ada Shannon] Treat it as an illustration. It rests on five tasks per occupation and a gap in value of about ten percent. The paper's own sensitivity check supports that caution. Switching the energy split from active-request shares to equal batch shares changes eight NVJ ranks and nine TSJ ranks across 13 categories, and moves the TSJ leader. Model rankings are sturdier, since the pairwise rank agreement across categories runs from 0.478 to 0.879. The extremes hold. The middle is soft.

61
00:12:19,023 --> 00:12:22,831
[Hal Turing] And everything is scored by a small judge, Gemini 3.1 Flash-Lite.

62
00:12:22,831 --> 00:12:52,088
[Dr. Ada Shannon] It matched human reviewers on 23 of 32 comparisons, 71.9%. That is a tiny sample. The three LLM judges agree with each other 72 to 77% of the time. Since value is zero for a loss and full for a win, the judge decides everything. The paper doesn't report a re-ranking under each judge in the main text. I'd want to know whether judge error tracks model family or output length, because a bias toward short formulaic answers would flatter exactly the cheap models.

63
00:12:52,088 --> 00:12:54,967
[Hal Turing] Now the human baseline. That's the one that moved the most.

64
00:12:54,967 --> 00:13:33,141
[Dr. Ada Shannon] Against an expert using AI, GLM's NVJ drops from 168.1 to 50.0, about seventy percent. Experts with AI spend roughly 23% less time, and GLM matches them about half as often. The caveats are real. The assisted references used Claude in a chat window, and different annotators made them. Retries show the other side. Four attempts per task raise value to $93.42, but energy goes from 0.43 to 1.76 megajoules and NVJ falls to 48.6. The denominator pays for every try, and the numerator only counts a win.

65
00:13:33,141 --> 00:13:36,391
[Hal Turing] Back to the prior art. Does this actually beat Cost-of-Pass?

66
00:13:36,391 --> 00:14:04,395
[Dr. Ada Shannon] The one test case is GPT-OSS 120B versus GLM. Their dollar cost and wall time are within a few percent, so a dollar-based view calls them a tie. Energy differs by 68%. But the paper never runs Cost-of-Pass or reports a rank correlation, so how often the two diverge is unknown. And value per AI dollar and per AI hour still rank GLM above GPT-OSS 120B, while NVJ ranks it below.

67
00:14:04,395 --> 00:14:06,949
[Hal Turing] What would you question about the 'value' itself?

68
00:14:06,949 --> 00:14:38,110
[Dr. Ada Shannon] It's expert time times median wage, credited only on a win. That's a proxy, not GDP. Using mean wages instead lifts GLM's NVJ to 194.6. Energy leaves out facility overhead, and MLPerf Power, by Tschand and colleagues in 2024, shows what whole-system metering catches that GPU counters miss. Nothing prices the expert reviewing outputs either. A cheap model with a low win rate generates more outputs to screen.

69
00:14:38,110 --> 00:14:39,782
[Hal Turing] What does this say about workers?

70
00:14:39,782 --> 00:15:17,538
[Dr. Ada Shannon] Less than headlines will claim. In some categories the AI beats a lone expert often. Against an assisted expert the margin is much smaller. Nothing here measures displacement, and Acemoglu's 2024 Simple Macroeconomics of AI argues task gains translate into modest aggregate effects. The Remote Labor Index, by Mazeika and colleagues in 2025, found far lower automation rates on messy freelance projects. The authors point to GDPWorks, 83 longer tasks with a median of 4.0 expert hours against GDPval's 2.34, as harder ground.

71
00:15:17,538 --> 00:15:18,884
[Hal Turing] Where should this go next?

72
00:15:18,884 --> 00:15:41,175
[Dr. Ada Shannon] Run each model at its minimal sufficient GPU count. Add seed repeats. Use expert graders instead of the small judge. Meter facility overhead. Then test the proposed uses, routing and stopping, which are only suggested here. Also try data-parallel replicas, since the 'more GPUs waste energy' result is a tensor-parallel result.

73
00:15:41,175 --> 00:15:42,336
[Hal Turing] So, where do we land?

74
00:15:42,336 --> 00:15:50,231
[Dr. Ada Shannon] The paper establishes a framework that makes energy visible, and an observation that joules, dollars and time can rank the same systems differently.

75
00:15:50,231 --> 00:16:02,724
[Hal Turing] And the boundary: the specific rankings and ratios are confounded by hardware, rest on single runs and a noisy small judge, and come from one task set. Thanks for listening, everyone.