PODCAST COMPANION · INTERACTIVE VISUALS

Code Predicts, Jev Judges: Separating Simulation from Evaluation

Paper: "Code Owns the Simulation, Jev Owns the Evaluation" — Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang · 2026 preprint · one closed API-only model (jev-1.13.0), single call, no reasoning text
arXiv 2610.01834 Interactive page

The pipeline Schematic

Toggle who does the predicting.

Five predictions of the paper

Hover or tap a row for the reported evidence.
Pick a claim.

Evaluation vs simulation Schematic

The label comes from how the task is presented, never from Jev's behavior.

3×3 payoff game Illustrative game

Payoffs are made up to show iterated elimination. Row's payoff is listed first, Column's second.

Accuracy by condition Reported

210 random games, 30 per depth-and-alignment cell, 3 option orders. Chance = 1/3.
Hover a bar.

Can Jev predict at all? Reported

The 90 misleading games, separate-call test.
Hover a bar.

Does its own prediction guide its choice? Reported

Plain one-call accuracy, split by whether Jev's separately asked prediction was right.
Hover a bar. The 35% rests on 26 calls and has no interval in the paper.

Step types, seen games Reported

Expert replays, steps labeled in advance. Shortcut: command mentions an object or place from the task text.
Hover a bar.

The butterknife step Illustrative

Task: put a clean butterknife in drawer. Jev holds a dirty knife.

Phrasing sensitivity Reported

Code forks the game Reported

Unseen games solved, out of 134. Diamonds mark the second runs.

What a lure tells you Schematic

A fixed shortcut, chosen in advance, sorts simulation decisions in two.
Hover a quadrant.

Control: trap questions Reported

150 CRT-style items, four listed options each. Interval 0.97–1.00 on the 99%.
Hover a bar. Listed options turn the task into checking each answer, so humans' open-ended 55% is not a like-for-like comparison.

Stacker robot (DeepMind Control Suite) Reported

Code supplies skills; inverse kinematics and PD control sit under each option.

Where intervals exist Per the discussion

Hover a cell.
Hover an item.

Gap vs noise Reported

Repeated identical calls: probabilities shift 0.04 on average (max 0.12) and the chosen option flips in 12% of cases.
Hover a bar.

Wide intervals Reported

Hover a row.

Missing comparisons

Three matched baselines that would show whether the boundary is Jev's or general.
Hover a box.

References

    Only arXiv 2610.01834 is given as an arXiv ID. The other sources link to scholar searches.