1
00:00:01,000 --> 00:00:07,733
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,733 --> 00:00:54,591
[Hal Turing] Today we're looking at "MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces." The first author is Srinivas Sridharan, listed at NVIDIA, with Tushar Krishna from Georgia Tech as the other lead. The author list spans NVIDIA, Keysight, AMD, Meta, Scala Computing, MLCommons, Georgia Tech, Harvard and HPE, among others. It's an MLSys 2026 paper, posted to arXiv in May 2026. The question is whether one open, graph-based trace format can let the industry observe real distributed AI jobs, replay them as benchmarks, and drive simulators for future hardware, all without exposing anyone's proprietary model.

3
00:00:54,591 --> 00:01:27,053
[Dr. Ada Shannon] The pitch is that today every tool in that chain speaks its own private workload dialect. Simulators, emulators and replay tools each have a format. So a bottleneck found in production gets lost or distorted before it reaches the people designing the next chip. Chakra wants to be the shared format in the middle. It began as a Meta-led effort with Georgia Tech in early 2023, and MLCommons adopted it. It's now a working group with over forty members. I'll say up front that it's a strong ecosystem story. Whether the science backs the strongest claims is something we'll have to look at carefully.

4
00:01:27,053 --> 00:01:30,814
[Hal Turing] Why can't people just share the model? Or at least a description of it?

5
00:01:30,814 --> 00:02:04,344
[Dr. Ada Shannon] Hyperscalers won't. A faithful description reveals architecture, parallelism choices and cluster topology. So what circulates today is often a spreadsheet of parameters under NDA, and a spreadsheet loses scheduling, overlap and dependency detail. The public alternative is MLPerf, from MLCommons. It's a real yardstick, but it evolves slowly, and a fixed suite invites over-optimization to the suite. The third option is running the full stack yourself, which needs an expensive cluster and expertise across several domains.

6
00:02:04,344 --> 00:02:05,737
[Hal Turing] So where does Chakra sit?

7
00:02:05,737 --> 00:02:39,592
[Dr. Ada Shannon] Picture a loop with four stages. Observe production. Reproduce the problem with benchmarks. Design and evaluate the next platform in simulators or emulators. Deploy, then observe again. Chakra's claim is that the same workload description can travel through every stage. The analogy is instruction traces and memory traces in CPU design, where simulators like gem5 are driven by traces of real programs. It's an analogy, not evidence that it works here. The AI version is harder because the network often matters as much as the chip.

8
00:02:39,592 --> 00:02:42,843
[Hal Turing] Okay, so what is the thing itself? What's an execution trace?

9
00:02:42,843 --> 00:03:08,059
[Dr. Ada Shannon] A directed acyclic graph of one device's work. Nodes are operations: compute, memory, or communication. Edges are dependencies: control, data, and synchronization. Nodes carry tensor shapes and optionally recorded durations. They carry no weights and no data. And there's one trace per GPU or NPU. You can't train from it. You can ask what happens to step time if bandwidth doubles.

10
00:03:08,059 --> 00:03:10,707
[Hal Turing] Sounds like a compiler IR, or a packet capture.

11
00:03:10,707 --> 00:03:35,413
[Dr. Ada Shannon] Both fit, with limits. Like an IR, it decouples front ends from back ends. Like a pcap, it records behavior without the application. But a pcap is a flat record, and this carries the dependency structure. Compared with a torch.fx graph, which exists to compile the model, this describes one observed run on one configuration, including the communication the parallelism strategy inserted.

12
00:03:35,413 --> 00:03:38,292
[Hal Turing] How do the parallelism strategies show up in the graph?

13
00:03:38,292 --> 00:03:58,400
[Dr. Ada Shannon] Through process groups. A communication node names which group of ranks takes part, so tensor, pipeline, data and expert parallelism are all expressible. The node set is deliberately small, with extensible attributes. There are protobuf and JSON formats. Timing fields, the start and the duration, are optional hints.

14
00:03:58,400 --> 00:04:01,140
[Hal Turing] Where do traces come from? You said recorded durations.

15
00:04:01,140 --> 00:04:24,778
[Dr. Ada Shannon] That's the distinction to hold onto. A post-execution trace is recorded from a real run. It bakes in that machine's parallelism, its optimizations and its measured timings. A pre-execution trace comes from a synthetic generator or a compiler. It isn't tied to one machine, and it's the only kind that could describe hardware that doesn't exist yet. But it needs timing from somewhere else.

16
00:04:24,778 --> 00:04:29,701
[Hal Turing] So the realistic trace has timings from the wrong machine, and the portable one has no timings.

17
00:04:29,701 --> 00:04:35,413
[Dr. Ada Shannon] Yes. Hold that thought, because it comes back when we ask what the paper can actually predict.

18
00:04:35,413 --> 00:04:38,014
[Hal Turing] How does a real trace get captured from PyTorch?

19
00:04:38,014 --> 00:04:58,865
[Dr. Ada Shannon] Two profilers, each seeing half the picture. The execution graph observer records the host side: CPU operators and kernel launches. Kineto records the device side: the kernels themselves with microsecond timing. Neither gives a full dependency graph alone, so Chakra merges them. The mechanics are for later.

20
00:04:58,865 --> 00:05:04,670
[Hal Turing] And once you have traces, there are three ways to use them, right? Replay, simulation, emulation.

21
00:05:04,670 --> 00:05:26,032
[Dr. Ada Shannon] Replay re-executes the recorded operators on a real system, through PyTorch's ATen and the c10d communication layer, so it exercises real hardware and the real collective library. Simulation models time in software, which is what ASTRA-sim and Scala's tool do. Emulation, which is Keysight's approach, generates real traffic against physical hardware under test.

22
00:05:26,032 --> 00:05:28,262
[Hal Turing] ASTRA-sim comes up a lot. Is that new here?

23
00:05:28,262 --> 00:05:41,822
[Dr. Ada Shannon] No. ASTRA-sim is the Georgia Tech simulator from Rashidi and colleagues in 2020. A 2.0 version by Won and colleagues followed in 2023. So the format and its main consumer share a lineage.

24
00:05:41,822 --> 00:05:47,952
[Hal Turing] One more term before we go on. The paper breaks step time into computation, exposed communication, and idle time.

25
00:05:47,952 --> 00:06:05,228
[Dr. Ada Shannon] Exposed communication is communication time not hidden behind compute. Idle time is the gaps between kernels: CPU launch overhead, host stalls, waiting on stragglers. Keep both in mind. Whether the trace keeps or drops that idle time turns out to matter a lot.

26
00:06:05,228 --> 00:06:12,333
[Hal Turing] So the observer gives you the host-side graph and Kineto gives you device timings. How do those two get stitched into one graph?

27
00:06:12,333 --> 00:06:47,163
[Dr. Ada Shannon] A linker does it. First, a PyTorch patch puts a shared identifier on both records, so a CPU operator and the GPU kernel it launched can be matched. Then the linker rebuilds three kinds of edges: control, data, and synchronization. The synchronization edges come from things like cudaStreamSynchronize and event record and wait. The design is inspired by Holistic Trace Analysis, and Lumos tackles a similar dependency problem. Then the converter checks that the result is acyclic, prunes redundant edges, and writes the standard trace.

28
00:06:47,163 --> 00:06:50,182
[Hal Turing] And the pre-execution side, the hypothetical hardware side?

29
00:06:50,182 --> 00:07:08,897
[Dr. Ada Shannon] The paper only surveys it. Generators like STAGE, SimAI, AMD's analytical framework, and Flint, which works from torch.fx graphs, build traces from a model description. LayerDAG is aimed at obfuscating intellectual property. Notice that none of them has measured timings from a real run.

30
00:07:08,897 --> 00:07:10,987
[Hal Turing] What do you do with a trace once you have it?

31
00:07:10,987 --> 00:07:49,810
[Dr. Ada Shannon] There's a visualizer, and a reconstructor that schedules the graph with a topological sort, Kahn-style. The important piece for simulators is the ET feeder. It streams nodes in memory-bounded windows and releases a node only when all its parents are done. A scheduling policy picks only among ready nodes, so it can't break a dependency. Replay runs one process per rank. It can do compute only, comms only, or the full thing, and it can replay a subtrace. Tensors can be pre-allocated or lazy, and inputs are randomized for privacy. It runs on any device that supports the recorded operators.

32
00:07:49,810 --> 00:07:53,572
[Hal Turing] Okay, results. Does the reconstruction actually match what Kineto saw?

33
00:07:53,572 --> 00:08:18,324
[Dr. Ada Shannon] Computation and exposed communication line up closely, across GPT3, Llama3, Mixtral and DeepSeek. But Chakra deliberately leaves out the idle time between kernels. The paper never says how large that omitted piece is, and it gives no numeric error, only the stacked bars. And since the reconstruction reuses recorded durations, matching computation mostly shows the tooling preserved the recording.

34
00:08:18,324 --> 00:08:21,854
[Hal Turing] So the parts it keeps agree. We just can't tell how far off the whole step is.

35
00:08:21,854 --> 00:08:22,411
[Dr. Ada Shannon] Right.

36
00:08:22,411 --> 00:08:24,455
[Hal Turing] The bandwidth result looked cleaner.

37
00:08:24,455 --> 00:09:12,427
[Dr. Ada Shannon] It's measured on Mixtral 8x22B across 32 H200s. Cut InfiniBand bandwidth by four, and All-to-All takes about 4.1 times longer and AllGather about 4.4 times. All-to-All is the exchange where every GPU sends a different piece to every other GPU. AllGather collects everyone's piece onto every GPU. Both are bandwidth-hungry, so slowing by roughly the bandwidth ratio is the naive expectation. The figure has two more labels. AllReduce fits the smaller slowdown at 1.5x, which the text explains by lower latency sensitivity and smaller volume. The 9.7x label goes unexplained. And the two bandwidths may come from two different clusters, with no error bars.

38
00:09:12,427 --> 00:09:15,167
[Hal Turing] So it attributes time well, on hardware that exists.

39
00:09:15,167 --> 00:09:45,632
[Dr. Ada Shannon] Yes. Replay is the other measured piece. On a 43-billion-parameter Megatron run, they compared the top ten NCCL kernels. Nine ran faster in replay, by up to about 29 percent. One ran about 22 percent slower. The authors blame missing memory contention, because replay isolates communication from the compute it normally fights with. Ten kernels, one workload. A vendor also reports a 2x collective gain from replay, but with no baseline shown.

40
00:09:45,632 --> 00:09:48,046
[Hal Turing] What does the trace library look like in aggregate?

41
00:09:48,046 --> 00:10:12,892
[Dr. Ada Shannon] The same model gives very different traces under different parallelism. Llama3 70B with tensor parallel 8 has about 16,897 AllReduce operations per GPU. With tensor parallel 4 and pipeline parallel 2, it's 130. Most compute kernels last roughly 2 to 100 microseconds. Nodes often have tens to hundreds of data-dependency parents.

42
00:10:12,892 --> 00:10:15,307
[Hal Turing] Then the simulators. What did ASTRA-sim show?

43
00:10:15,307 --> 00:10:34,904
[Dr. Ada Shannon] A topology and bandwidth sweep, 8 GPUs, Mixtral 8x7B. Switch was best, then ring, then fully connected. Communication time stopped improving as bandwidth grew, because latency took over. The authors write that they suspect that explanation. There's no measured ground truth to check it against.

44
00:10:34,904 --> 00:10:36,251
[Hal Turing] And the other two tools?

45
00:10:36,251 --> 00:11:06,530
[Dr. Ada Shannon] Scala SCP simulated 256 A100s on a Llama-2 trace over a two-tier Clos with 400 gigabit NICs. Average link utilization stayed well below line rate, with long underused stretches. The Keysight emulation is the interesting one. Real traffic, four-switch Clos, 400 gigabit, DCQCN congestion control. Mixing All-Reduce with All-to-All makes All-Reduce time variable and creates long-tail All-to-All stragglers. But that traffic came from a synthetic trace.

46
00:11:06,530 --> 00:11:08,480
[Hal Turing] Real hardware, invented workload.

47
00:11:08,480 --> 00:11:12,010
[Dr. Ada Shannon] Yes. It's a hardware observation, with a synthetic input.

48
00:11:12,010 --> 00:11:13,264
[Hal Turing] Last piece: inference.

49
00:11:13,264 --> 00:11:52,180
[Dr. Ada Shannon] They extended capture to vLLM v1, and it's described as ongoing. For mixture-of-experts, routing nodes carry per-expert token counts, so imbalance is visible. Their example is Mixtral 8x7B on two expert-parallel ranks with six input tokens. For KV-cache offloading on Llama3-8B, device-to-host copies go from 387 to 5,958, and their GPU time from 0.895 milliseconds to 216.484 milliseconds. That's the number to remember. And prefill-to-decode KV transfer comes to roughly 110 to 190 microseconds per layer.

50
00:11:52,180 --> 00:12:07,738
[Hal Turing] So the tooling covers a lot. Whether any of it predicts hardware that doesn't exist yet is the part I want to push on. Start with the idle time Chakra drops. That's where launch overhead, host stalls and stragglers live. How much of a real step is that?

51
00:12:07,738 --> 00:12:31,329
[Dr. Ada Shannon] The paper doesn't say. There's no percent error, no per-workload number, and no idle fraction. What's established is that the pipeline preserves compute and exposed-communication structure. What's unknown is how much of the step was removed, and so how much the "closely aligned" claim leans on dropping the hardest part. It's also partly circular, because the reconstruction reuses the recorded durations.

52
00:12:31,329 --> 00:12:33,837
[Hal Turing] There's a wrinkle in the paper's own text, too, right?

53
00:12:33,837 --> 00:12:45,493
[Dr. Ada Shannon] Yes. Related Work says Chakra captures idle periods and compute-communication overlap. Section 5.1 says it excludes idle time. Those two can't both be the whole story.

54
00:12:45,493 --> 00:12:50,277
[Hal Turing] Then the bigger claim, projecting onto hardware that doesn't exist yet. Does the paper show that?

55
00:12:50,277 --> 00:13:20,555
[Dr. Ada Shannon] No, and this is the claim I'd hold hardest. Replay and reconstruction carry durations from the source machine. Projecting to a new GPU needs an outside compute-time model. The experiment that would settle it is simple: record on platform A, predict step time on platform B, then compare with a real run on B. The paper doesn't include it. The ASTRA-sim topology sweeps and the Scala study have no ground truth, and the authors write "we suspect" about the topology result.

56
00:13:20,555 --> 00:13:22,320
[Hal Turing] What should it be measured against?

57
00:13:22,320 --> 00:13:52,785
[Dr. Ada Shannon] ASTRA-sim 2.0, by the supplied references, was validated against real systems. Mystique, from Liang and colleagues at Meta in 2023, is the closest prior work: it replayed production traces as benchmarks and reported fidelity numbers. Chakra cites it without matching it. Lumos, Liang again, 2025, models the same stream dependencies the Chakra linker reconstructs. I won't quote their numbers, but each shows what an accuracy table looks like.

58
00:13:52,785 --> 00:13:58,497
[Hal Turing] The measured results are the safer ground, I assume. The bandwidth comparison, and the replay table.

59
00:13:58,497 --> 00:14:29,101
[Dr. Ada Shannon] Safer, but they support less than they seem to. They're real measurements, which is evidence for analysis and replay, not for simulation accuracy. Big collectives slowing a bit over four times for a quarter of the bandwidth is what a first-order model predicts. But the two bandwidths may come from two clusters, Georgia Tech and HPE. The unexplained 9.7x label sits beside a latency story that doesn't reconcile it. No error bars, so I wouldn't read anything into 4.4 versus 4.

60
00:14:29,101 --> 00:14:30,030
[Hal Turing] And replay?

61
00:14:30,030 --> 00:14:46,516
[Dr. Ada Shannon] Replay ran mostly faster than the original, which the paper blames on missing memory contention. That's useful for tuning communication libraries. But overlap with compute is part of why you'd replay a real step, and there's no full-step replay error here. The vendor's 2x collective gain has no setup or before-and-after.

62
00:14:46,516 --> 00:14:49,720
[Hal Turing] The Keysight congestion result looked like the most concrete insight.

63
00:14:49,720 --> 00:15:03,234
[Dr. Ada Shannon] It is. Mixed All-Reduce and All-to-All made All-Reduce variable and gave All-to-All a long tail under DCQCN. But the trace was synthetic, so I'd call it a credible stress test, not a finding about production jobs.

64
00:15:03,234 --> 00:15:04,441
[Hal Turing] What else is thin?

65
00:15:04,441 --> 00:15:25,200
[Dr. Ada Shannon] A post-execution trace bakes in the original parallelization and eager-mode capture. Per-device traces limit global scheduling. Traces reach gigabytes. The inference section is explicitly ongoing, and a six-token routing example shows capture, not replay. And the collective accuracy checker results are withheld as proprietary.

66
00:15:25,200 --> 00:15:26,779
[Hal Turing] So what do we actually credit?

67
00:15:26,779 --> 00:15:45,401
[Dr. Ada Shannon] A real, adopted standard. PyTorch and NeMo can collect it, ASTRA-sim, Keysight and Scala consume it, there's a public trace library, and an MLCommons working group. It's also a plausible way to share workload structure without sharing the model. That's a genuine contribution, and a different one from accuracy.

68
00:15:45,401 --> 00:15:47,352
[Hal Turing] So what can a practitioner trust today?

69
00:15:47,352 --> 00:16:09,411
[Dr. Ada Shannon] Cross-tool workload exchange, trace-based bottleneck attribution, communication replay and tuning, and traffic emulation against switches and NICs. Absolute step-time prediction on future hardware needs independent validation, with idle time included and errors reported per workload. The authors point to compression, InfraGraph and MLPerf Storage next.

70
00:16:09,411 --> 00:16:21,671
[Hal Turing] Then the summary is this. Chakra establishes a shared, working format that many tools can read. It doesn't yet establish that those tools predict real hardware. Thanks for listening, everyone.