1
00:00:01,000 --> 00:00:08,244
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:08,244 --> 00:00:39,312
[Hal Turing] Today we're on "Intelligence per Watt: Measuring Intelligence Efficiency of Local AI," by Jon Saad-Falcon, Avanika Narayan, and colleagues, from Stanford University and Together AI. It first appeared on arXiv in November 2025, and the latest version is dated October 2026. The question is whether small models on laptop-class hardware can take real load off the cloud, and whether a new metric can tell us how fast that's becoming true.

3
00:00:39,312 --> 00:00:55,984
[Dr. Ada Shannon] The stakes are practical. Cloud demand for LLM queries is straining infrastructure. If a large share of ordinary queries can be answered on the device in front of you, that changes where the compute and the power bill land. The paper claims to measure that, and the measuring is where the care is needed.

4
00:00:55,984 --> 00:00:58,074
[Hal Turing] They open with a historical analogy, right?

5
00:00:58,074 --> 00:01:27,146
[Dr. Ada Shannon] Yes. Koomey and colleagues showed performance per watt doubling roughly every year and a half from 1946 to 2009. The paper's argument is that workloads moved from mainframes to PCs once efficiency fit a personal device's power limit, not because PCs beat mainframes on peak speed. So local inference becomes viable when efficiency meets the laptop power envelope. That's the authors' framing. It's an analogy, not evidence about LLMs.

6
00:01:27,146 --> 00:01:29,050
[Hal Turing] So first, what counts as "local"?

7
00:01:29,050 --> 00:02:08,756
[Dr. Ada Shannon] A model with at most 20 billion active parameters per forward pass. Active, not total, because per-query power and latency depend on the parameters touched for each token. That matters for mixture-of-experts models, which hold many expert sub-networks but route each token through only a few. You get large-model knowledge at small-model compute per token. Mixtral-8x7B is about 47 billion total and about 12.9 billion active. gpt-oss-120B has 120 billion total and at most 20 billion active. Both qualify. Frontier cloud models are at least 100 billion.

8
00:02:08,756 --> 00:02:10,753
[Hal Turing] But the total still has to fit in memory.

9
00:02:10,753 --> 00:02:27,146
[Dr. Ada Shannon] Right, and that's why hardware matters. A mixture-of-experts model needs memory room for every expert. A unified-memory machine like the Apple M4 Max shares one large pool between CPU and GPU, so it can hold models a 24 GB discrete GPU can't.

10
00:02:27,146 --> 00:02:30,490
[Hal Turing] Now the metric. Intelligence per watt. What's being divided?

11
00:02:30,490 --> 00:02:45,815
[Dr. Ada Shannon] Average task accuracy divided by average power draw in watts, taking the ratio of means across queries. Accuracy is binary per query: right or not. Power is the average draw during that query's inference. Speed doesn't enter the ratio at all.

12
00:02:45,815 --> 00:02:47,672
[Hal Turing] That last part sounds like trouble.

13
00:02:47,672 --> 00:03:16,186
[Dr. Ada Shannon] It's why the paper also defines intelligence per joule. Energy per query is power times latency. A watt is a rate; a joule is a bill. Picture a device that sips power but takes ten times as long. Its watts look great, but the battery pays for every second it stays busy, so its joules can be worse. The analogy has a boundary: real devices also have idle draw and startup costs. The paper headlines per-watt, but reports four variants, average and peak, per watt and per joule.

14
00:03:16,186 --> 00:03:20,134
[Hal Turing] What does "accuracy" mean on open-ended chat, where there's no answer key?

15
00:03:20,134 --> 00:03:44,190
[Dr. Ada Shannon] For the benchmarks with ground truth, MMLU-Pro, SuperGPQA, and NaturalReasoning, it's exact match. For real chat from WildChat, a judge model, Qwen3-235B, compares the local answer to a reference answer that is also from Qwen3-235B. A local win or tie counts as success. So chat accuracy is relative to one particular model's answer, which matters later.

16
00:03:44,190 --> 00:03:45,444
[Hal Turing] And "best-of-local"?

17
00:03:45,444 --> 00:03:58,633
[Dr. Ada Shannon] A query counts as covered if any of the 20-plus local models answers it correctly. That selection uses the ground truth, so it's an oracle router, not something you could deploy. Keep that in mind whenever a coverage number comes up.

18
00:03:58,633 --> 00:04:02,719
[Hal Turing] One more definition, since you said power matters. What are they actually measuring?

19
00:04:02,719 --> 00:04:31,791
[Dr. Ada Shannon] Accelerator-only power: the AI chip subsystem, read through NVML on NVIDIA, ROCm SMI on AMD, and powermetrics GPU power on Apple Silicon. That excludes CPU, host memory, fans, and power-supply losses, which a wall-power meter would capture. And the main setting is batch size 1, one query at a time. That isolates model-and-chip efficiency and matches how a personal device is used, but it's conservative for cloud chips, which thrive on batching.

20
00:04:31,791 --> 00:04:33,370
[Hal Turing] So what's the study itself?

21
00:04:33,370 --> 00:04:56,079
[Dr. Ada Shannon] About a million single-turn queries from four sources: WildChat, NaturalReasoning, MMLU-Pro, and SuperGPQA. More than twenty models, eight accelerators, three questions: how much can local models cover, how has efficiency changed over time, and how much could routing save. Single-turn only, so no agents or multi-turn work.

22
00:04:56,079 --> 00:04:58,122
[Hal Turing] Is anyone else measuring LLM energy?

23
00:04:58,122 --> 00:05:44,051
[Dr. Ada Shannon] Three relevant works. Samsi and colleagues, "From Words to Watts," from 2023, was an early GPU study of LLM inference energy, and the paper compares its own sampling interval against theirs. Fernandez, Na, Tiwari, Bisk, Luccioni, and Strubell wrote "Energy Considerations of Large Language Model Inference and Efficiency Optimizations" in 2025, which the paper cites as precedent for energy measurement practice, and it notes that work's tooling samples more coarsely, every 15 seconds. And Tschand and colleagues' "MLPerf Power," from HPCA 2025, is the established standard, with wall-power methodology from microwatts to megawatts. The paper cites it but doesn't use it for measurement.

24
00:05:44,051 --> 00:05:46,977
[Hal Turing] So the comparison with MLPerf is worth holding onto.

25
00:05:46,977 --> 00:05:54,593
[Dr. Ada Shannon] It is. First, though, the numbers: how the harness works, and what the paper says happened between 2023 and 2025.

26
00:05:54,593 --> 00:05:59,376
[Hal Turing] Okay, the numbers. Start with the data. Why mix chat logs with benchmarks?

27
00:05:59,376 --> 00:06:20,785
[Dr. Ada Shannon] The mix answers two different questions. WildChat is real user chat, so it shows what people actually ask. NaturalReasoning, MMLU-Pro and SuperGPQA have answer keys, so correctness is unambiguous. WildChat has no key, so Qwen3-235B writes the reference and also judges, as we covered. The other three use exact match.

28
00:06:20,785 --> 00:06:22,596
[Hal Turing] And per query, the harness records what?

29
00:06:22,596 --> 00:06:48,974
[Dr. Ada Shannon] Accuracy, latency, time to first token, power, and energy. Energy comes from sampling power every 50 milliseconds and numerically integrating over the query. Multi-GPU runs sum each GPU rather than scaling up from one. vLLM serves everything, at batch size one. The authors also say software power readings can be off by 10 to 15 percent, and they claim consistent relative comparisons, not absolute accuracy.

30
00:06:48,974 --> 00:06:53,014
[Hal Turing] So back to the metric. Where does the headline five-fold gain come from?

31
00:06:53,014 --> 00:07:17,674
[Dr. Ada Shannon] Table 2 follows one best local pairing per year. In 2023 it's Mixtral-8x7B on a Quadro RTX 6000. In 2024, Llama-3.1-8B on an RTX 6000 Ada. In 2025, gpt-oss-120B on an Apple M4 Max. Success rate climbs from about 23 queries in a hundred, to 49, to 71. Accuracy per watt grows 5.3 times overall, so about five-fold.

32
00:07:17,674 --> 00:07:24,547
[Hal Turing] Then the paper says 3.1 times from models and 1.7 from hardware. Where did each number come from?

33
00:07:24,547 --> 00:07:50,693
[Dr. Ada Shannon] It's bookkeeping. The 3.1 is just the accuracy ratio, 71.3 over 23.2. Divide the total 5.3 by that and you get roughly 1.7. That leftover is labeled hardware. But the leftover is a power ratio, so it implies average power fell about 1.7 times, from roughly 290 watts to roughly 170. The paper doesn't state those wattages. That's my arithmetic.

34
00:07:50,693 --> 00:07:53,665
[Hal Turing] So nobody ran the same model on two chips to get the 1.7.

35
00:07:53,665 --> 00:08:10,058
[Dr. Ada Shannon] Not in this table. Each year changes model and chip together. I'll hold the critique, but that's the setup. The error bars are one standard deviation across measurement runs. No significance test is reported, and the paper doesn't say whether the spread includes query sampling.

36
00:08:10,058 --> 00:08:13,588
[Hal Turing] Fine. Now coverage. What does one small model manage on its own?

37
00:08:13,588 --> 00:08:36,436
[Dr. Ada Shannon] Averaged over the four datasets, single local models run from about 50 queries in a hundred for Qwen3-4B up to about 71 for gpt-oss-120B. No single small model gets past that. But the paper also builds a pool of more than twenty local models and counts a query as covered if any one of them gets it right. That reaches 88.7 percent.

38
00:08:36,436 --> 00:08:37,736
[Hal Turing] Any one of them. Chosen how?

39
00:08:37,736 --> 00:08:59,377
[Dr. Ada Shannon] By checking the answer key, so it's an oracle. Nobody can deploy that. It's an upper bound on what perfect routing among those models could cover. It does say something real, though: different models fail on different queries. The paper even notes the pool beats the best cloud model on three of four benchmarks, but it attributes that to picking the best of twenty-plus versus three.

40
00:08:59,377 --> 00:09:01,606
[Hal Turing] Is chat easier than reasoning, as you'd guess?

41
00:09:01,606 --> 00:09:25,987
[Dr. Ada Shannon] Yes. The best local model covers about 89 of every hundred WildChat queries but about 65 on NaturalReasoning. Coverage drops to about 60 percent in technical fields like architecture and engineering. And chat success means winning or tying against Qwen3-235B, scored by Qwen3-235B itself, which is a looser bar than an answer key.

42
00:09:25,987 --> 00:09:28,913
[Hal Turing] And the routing simulation? What's it routing against?

43
00:09:28,913 --> 00:09:50,740
[Dr. Ada Shannon] Against a baseline of sending everything to Qwen3-235B on one H200 at batch size 16. The workload is 80 million queries over 24 hours, from a public trace. Four small models run locally on an M4 Max. Misrouted queries fall back to the big model, so the paper says answer quality is kept.

44
00:09:50,740 --> 00:09:53,526
[Hal Turing] How good does the router have to be for the savings to show up?

45
00:09:53,526 --> 00:10:15,910
[Dr. Ada Shannon] An 80-percent-accurate router cuts energy by about 64 percent and cost by about 59. Even a 60-percent router saves roughly 45 to 48 percent. Perfect routing is the ceiling, at about 80 percent less energy. In the simulation the local models handle about 81 percent of queries. That's a different number from the 88.7, so don't merge them.

46
00:10:15,910 --> 00:10:18,139
[Hal Turing] Does that come from the laptop being efficient?

47
00:10:18,139 --> 00:10:35,090
[Dr. Ada Shannon] The paper says no. The dominant effect is using smaller models, and the baseline is already a batched cloud setup. Also, those savings assume a router that is right 60 or 80 percent of the time. No real router was built or tested in the study.

48
00:10:35,090 --> 00:10:37,923
[Hal Turing] Anything in the appendix that changes how I read this?

49
00:10:37,923 --> 00:11:02,397
[Dr. Ada Shannon] Two things, as the paper reports them. Each step down in precision to FP4 saves roughly three to three and a half times the energy, which matters because gpt-oss ships in FP4. And swapping vLLM for SGLang or llama.cpp shifts absolute IPW by 3 to 12 percent, but rankings hold, with Kendall tau around 0.9. So the ordering looks stable even if the absolute value isn't.

50
00:11:02,397 --> 00:11:21,623
[Hal Turing] Stable ordering, shaky absolute values. Now I want to poke at that five-fold. Each year changes the model, the chip, and the active parameter count together. Is the 3.1 times model, 1.7 times hardware split a real decomposition?

51
00:11:21,623 --> 00:11:43,078
[Dr. Ada Shannon] No. It's an attribution, not a measurement. The factors multiply, so the shares depend on the order you peel them off. Change the chip first and the model second, and you get different shares. The interaction term is invisible. And the hardware factor is just whatever is left after dividing out the accuracy ratio, so it's really a power ratio.

52
00:11:43,078 --> 00:11:49,162
[Hal Turing] But the harness covers eight accelerators and twenty-plus models. The raw material for a proper grid is there.

53
00:11:49,162 --> 00:11:58,914
[Dr. Ada Shannon] Right. Same model on all three chips, and all three models on one chip, would settle it. The source doesn't report that. Table 2 doesn't use the grid.

54
00:11:58,914 --> 00:12:06,019
[Hal Turing] Something in the hardware table bothers me. The 2025 chip, the M4 Max, has the lowest memory bandwidth of the three.

55
00:12:06,019 --> 00:12:52,041
[Dr. Ada Shannon] Checking Table 7: 546 gigabytes per second, against 672 for the 2023 card and 960 for the 2024 card. Decoding is usually bandwidth-bound. Now the per-joule decomposition: 18 times overall, with 5.9 times attributed to hardware. Divide that by the per-watt hardware figure and queries finish about three and a half times faster. A slower-memory chip winning on speed points to the model: fewer active parameters per token, FP4, a different serving path. That's my inference from the tables, not a measured result. There's also a small inconsistency: the Figure 3 caption says 3.0 times and sixteen months, the text says 3.1 and two years.

56
00:12:52,041 --> 00:12:55,013
[Hal Turing] So which metric answers whether local is becoming viable?

57
00:12:55,013 --> 00:13:17,026
[Dr. Ada Shannon] Per joule, because a user pays in battery and waiting time. Per watt is useful when a thermal envelope is the limit. But accuracy per watt has no time term, so a device that draws little and runs slowly can outrank a fast one. Classic performance per watt, the Koomey analogy, is really operations per joule, because performance is a throughput. That time term is exactly what this version drops.

58
00:13:17,026 --> 00:13:23,156
[Hal Turing] The paper reports both, and says rankings hold across all four variants. If rankings hold, why care?

59
00:13:23,156 --> 00:13:39,828
[Dr. Ada Shannon] Because the headline magnitude and the attribution flip. Per watt says hardware helped modestly. Per joule says hardware dominated. The source gives no reason for headlining per watt, and the abstract leads with 5.3 times while 18 times sits in a footnote and a figure.

60
00:13:39,828 --> 00:13:45,261
[Hal Turing] What about the measurement itself? Fifty-millisecond sampling beats Samsi and colleagues' 100.

61
00:13:45,261 --> 00:14:17,723
[Dr. Ada Shannon] It does, but the problem is scope, not cadence. This paper reads GPU-subsystem power only: NVML on NVIDIA, powermetrics GPU power on Apple. DRAM, CPU, fans and power supply losses are outside it. The paper itself cites 10 to 15 percent software error, drawing on Yang and colleagues in 2023. And whether Apple's GPU-only number and NVIDIA's board-level number are like-for-like is asserted, not shown. Unified memory may flatter the Mac.

62
00:14:17,723 --> 00:14:19,534
[Hal Turing] Compare that with MLPerf Power.

63
00:14:19,534 --> 00:14:46,841
[Dr. Ada Shannon] MLPerf Power uses audited wall-power scope, cross-vendor comparability, and quality targets as pass-or-fail gates rather than a ratio. A ratio hides thresholds: a model with 20 percent accuracy at tiny power could score well. Credit where due: MLPerf doesn't tie real-query accuracy to power in one number, and this paper releases an open harness. That's useful. It's still one lab's harness, unaudited.

64
00:14:46,841 --> 00:14:52,646
[Hal Turing] Now cloud versus local. The paper says the best cloud chip is only 1.4 times better per watt.

65
00:14:52,646 --> 00:15:16,330
[Dr. Ada Shannon] B200 over the M4 Max: 1.4 times per watt, 1.6 to 2.3 times per joule. A SambaNova SN40L reaches 6.5 to 7.4 times per joule. And that's batch size one, where cloud chips are handicapped. At batch 64 the B200 gets 11 to 20 times better per joule than at batch one. So 1.4 is not the practical gap.

66
00:15:16,330 --> 00:15:17,584
[Hal Turing] And the routing savings?

67
00:15:17,584 --> 00:15:41,918
[Dr. Ada Shannon] The paper itself says the dominant effect is smaller models, not local hardware. The simulation compares small models at batch one on a Mac against a batch-16 Qwen3-235B on an H200, and the source doesn't report a baseline where those same small models run on a batched cloud GPU. So it shows routing to small models saves energy. It doesn't show that running them locally is what does the saving.

68
00:15:41,918 --> 00:15:43,823
[Hal Turing] And the 88.7 percent coverage?

69
00:15:43,823 --> 00:16:05,138
[Dr. Ada Shannon] An oracle over twenty-plus models, selected using ground truth. The deployable single-model number is about 71 percent. For chat, success is win-or-tie against Qwen3-235B, judged by Qwen3-235B. That's a lenient criterion, and the source doesn't give me a sensitivity analysis with other judges.

70
00:16:05,138 --> 00:16:07,135
[Hal Turing] So what do I take away as a practitioner?

71
00:16:07,135 --> 00:16:42,662
[Dr. Ada Shannon] Evidence supports three things: a lot of single-turn chat and knowledge queries are locally answerable, MoE on memory-rich devices does well, and hybrid routing saves resources in simulation. It does not give you a controlled hardware-versus-model split, absolute energy figures, or real router accuracy. The paper's multi-turn extension reportedly confirms the patterns, but it sits outside the main study, so I wouldn't lean on it for agent workloads at scale. Closing the gaps takes a factorial design, wall-power metering, quality-threshold gating, and per joule as a co-headline.

72
00:16:42,662 --> 00:17:03,606
[Hal Turing] So: local models on local chips now answer a large share of ordinary queries, and joule-level efficiency improved a lot from 2023 to 2025. The boundary is that the model-versus-hardware split isn't established, and accelerator-only per-watt numbers aren't a cross-vendor energy standard. Thanks for listening, everyone.