1
00:00:01,000 --> 00:00:06,944
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:06,944 --> 00:00:40,334
[Hal Turing] Today we're looking at "Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?" It's an October 2026 arXiv paper from Huangfei Jiang, Anning Hu, Kai Sheng and colleagues at Prophetlab and RethinkLab, with SJTU, USTC, and the buy-side firms Foresight Fund and IvyRock Asset Management. The question is whether a benchmark can freeze every document and calculation at a decision time, and then tell us whether agents turn evidence into sound investment judgments.

3
00:00:40,334 --> 00:00:51,433
[Dr. Ada Shannon] The stakes: if an agent peeks at the future, its research score measures the leak, not the analyst. The authors say a date filter doesn't stop that, and they have a concrete example.

4
00:00:51,433 --> 00:00:53,477
[Hal Turing] The Tesla one. Walk me through it.

5
00:00:53,477 --> 00:01:14,467
[Dr. Ada Shannon] A task called O74 sets the decision time at March 24, 2026. A search with a before-March-24 filter still returned a source from July 2. That source reported realized second-quarter deliveries of 480,126. So the agent was asked to forecast a number that a returned page already contained.

6
00:01:14,467 --> 00:01:19,111
[Hal Turing] How does a date filter let that through? The filter said before March 24.

7
00:01:19,111 --> 00:01:44,560
[Dr. Ada Shannon] A date filter constrains the query, using the date attached to a page. But a page can keep its old publication date while its content gets updated. The label says March; the text says July. The paper's alternative is version-level availability. Every revision of a document is stored as its own version with its own availability time. A version is eligible only if that time precedes the decision time, which they call t0.

8
00:01:44,560 --> 00:01:46,418
[Hal Turing] And that applies beyond the search box?

9
00:01:46,418 --> 00:02:00,536
[Dr. Ada Shannon] Yes. Search, read, download, table tools and cached results all go through the same check. The database holds about 3.4 million document versions, 1.52 terabytes, across nine source categories.

10
00:02:00,536 --> 00:02:08,059
[Hal Turing] Before we go further, why is equity research hard to grade at all? Finance QA benchmarks already exist.

11
00:02:08,059 --> 00:02:31,976
[Dr. Ada Shannon] Those are mostly single-stage. FinanceBench, from Islam and colleagues at Patronus AI in 2023, asks questions over filings. Finance Agent, FinSearchComp, FinBench and FinanceGym add tools or multi-step work. The gap the authors claim is that none enforces version-level availability across the whole pipeline. I haven't verified that claim across every benchmark.

12
00:02:31,976 --> 00:02:36,898
[Hal Turing] Compare it to SWE-bench for me. There a unit test says pass or fail.

13
00:02:36,898 --> 00:03:03,230
[Dr. Ada Shannon] Right, and finance has no such oracle. The outcome, a realized price or earnings figure, arrives later and is noisy. A sound analysis can land on the wrong number, and a lucky guess can land on the right one. Equity research is a chain: evidence, then assumptions, then judgment. So the paper grades the chain with expert rubrics, in the style of OpenAI's HealthBench and GDPval, and not only the outcome.

14
00:03:03,230 --> 00:03:07,270
[Hal Turing] Define the rubric and the dependency graph, because that's where I'd get lost.

15
00:03:07,270 --> 00:03:41,450
[Dr. Ada Shannon] Each task has a private rubric of weighted criteria. Every criterion links to a source passage, a reproducible calculation, or an explicit financial judgment. The dependency graph connects facts, assumptions, intermediate calculations and conclusions. Take refinancing. Cash plus operating cash flow plus unused credit looks sufficient. But if a covenant makes that credit undrawable, the conclusion flips. The graph lets a grader see which link broke: missed evidence, bad arithmetic, a poor assumption, or inconsistent reasoning.

16
00:03:41,450 --> 00:03:43,400
[Hal Turing] So there are separate yardsticks, then.

17
00:03:43,400 --> 00:04:05,970
[Dr. Ada Shannon] Three. Answer quality, scored on rubric dimensions like evidence, financial logic, calculation and uncertainty. Forecast outcome, meaning criteria settled against what actually happened. And simulated return from a trading simulation. The paper's central claim is that these don't rank models the same way. We'll get to that.

18
00:04:05,970 --> 00:04:11,171
[Hal Turing] Now, the frozen database fixes the retrieval side. What about what the model already knows?

19
00:04:11,171 --> 00:04:30,026
[Dr. Ada Shannon] That's the second leakage channel, and I want to flag it now because the database doesn't touch it. Context leakage is information arriving through retrieved documents. Parametric leakage is what the model absorbed in pretraining. If a model was trained after t0, its weights may hold the outcome, and no database can remove that.

20
00:04:30,026 --> 00:04:32,441
[Hal Turing] Is that a known problem in finance specifically?

21
00:04:32,441 --> 00:05:08,757
[Dr. Ada Shannon] Well documented. Glasserman and Lin, 2023, studied GPT sentiment signals for stock returns. They anonymized company names in headlines, and the predictive power depended on whether the model could identify the company. The paper cites that one. Sarkar and Vafa, 2024, formalized lookahead bias in pretrained models and argued it's inherent to out-of-sample prediction with them. And He, Lv, Manela and Wu, 2025, built chronologically consistent models, each trained only on data up to its date. We'll come back to what the paper does with these.

22
00:05:08,757 --> 00:05:11,682
[Hal Turing] Fine, park it. What's the benchmark's actual shape?

23
00:05:11,682 --> 00:05:29,469
[Dr. Ada Shannon] Eighty-six expert-written assignments modeled on buy-side work: five task categories, plus Integrated Research, which chains the stages, and Market Trading. Eleven agent systems run them with the same tool library. Cutoffs fall roughly between March and August 2026.

24
00:05:29,469 --> 00:05:34,763
[Hal Turing] Let's start with the setup. What does an agent actually get to work with inside this frozen world?

25
00:05:34,763 --> 00:06:10,290
[Dr. Ada Shannon] Six tools, shared by every model. Search finds eligible document versions. Read returns passages and tables. Download pulls source files into a workspace. Table operations query structured data and freeze the result at retrieval, so a later revision can't change numbers already pulled. Bash does the number crunching with the network disabled. And submit_answer checks the deliverable format. Each task gets 7,200 seconds and up to three submissions. The cutoff is timezone-aware, and a record with an unverifiable timestamp is excluded rather than guessed at.

26
00:06:10,290 --> 00:06:14,934
[Hal Turing] Who writes the tasks? I'm guessing not a grad student with a finance textbook.

27
00:06:14,934 --> 00:06:59,423
[Dr. Ada Shannon] Buy-side practitioners. An expert frames the question and builds the dependency graph. Then AI models hunt for missing evidence and propose counterexamples, but the expert decides what gets adopted. Two refinements show how it works. For AWS O15, a model's revenue interval missed the realized number but earned 15.8 points just for being narrow. So the experts specified a central 80 percent interval and scored accuracy and width jointly. The revised interval missed again and got zero credit. For Microsoft O58, they added a criterion that catches double counting. Cloud revenue overlaps the three reporting segments, so adding it as a fourth segment is a scope error with its own test.

28
00:06:59,423 --> 00:07:00,816
[Hal Turing] And the scoring itself?

29
00:07:00,816 --> 00:07:24,454
[Dr. Ada Shannon] Weighted criteria per task, scaled to 100, with every task counting equally. Numeric criteria use deterministic checks. Semantic criteria go to GPT-6 Astra as judge. There are two modes. Answer-based grades the final available answer. Strict requires an accepted submit_answer call. The gap between them measures submission reliability, not reasoning.

30
00:07:24,454 --> 00:07:27,519
[Hal Turing] So the headline. Who wins, and what's the one thing to remember?

31
00:07:27,519 --> 00:07:55,987
[Dr. Ada Shannon] Astra leads at 68.1 and Opus is close behind at 66.9. But look inside Astra. It scores about 79 on Research & Evidence and about 78 on Evaluate & Decide. On Forecast & Model it scores 52.5. That's a gap of roughly 26 points inside one model. Opus has the same shape, 52.6 on forecasting. The authors' reading is that the bottleneck has moved from finding information to calibrating assumptions.

32
00:07:55,987 --> 00:08:01,327
[Hal Turing] Is that a weakness of the models, or just forecasting being hard? Even good analysts miss quarters.

33
00:08:01,327 --> 00:08:28,866
[Dr. Ada Shannon] The authors concede part of that. Forecast tasks are partly settled against realized outcomes, which include market noise. A sound analysis can still miss. So the 52.5 mixes model weakness with irreducible noise, and the paper can't separate the two. It does report that KPI forecasting is much lower still, 41.1 for Astra. No confidence intervals or seed variance accompany any of these differences.

34
00:08:28,866 --> 00:08:30,956
[Hal Turing] You said the yardsticks disagree. Show me.

35
00:08:30,956 --> 00:08:53,665
[Dr. Ada Shannon] On forecast-outcome criteria alone, DeepSeek V4.1 Flash leads, slightly ahead of Astra and Opus, which are nearly tied behind it. DeepSeek V4 Pro tops the Market Trading category even though it's mid-table overall. And in the simulated trading sample, Kimi K3 is eighth on research score, yet it returned 7.15 percent against 6.39 for Opus.

36
00:08:53,665 --> 00:08:57,102
[Hal Turing] That's a 0.76-point gap. I wouldn't hang a ranking on that.

37
00:08:57,102 --> 00:09:23,480
[Dr. Ada Shannon] Right, and the paper doesn't give you grounds to. The sample is twenty assignments, eleven companies, thirteen disclosures, over a few months, with shared disclosure dates and no variance reported. Every model beat always-long, which returned 1.18 percent. But with events that clustered, that mostly tells you the sample was kind. What survives is the weaker claim: research score and trading return aren't the same measurement.

38
00:09:23,480 --> 00:09:25,384
[Hal Turing] Now the case study. The AEP one.

39
00:09:25,384 --> 00:09:58,913
[Dr. Ada Shannon] The question: does stronger operating cash flow reduce the utility's funding need? Operating cash inflow rose by 750 million dollars, from about 2.7 billion to 3.4 billion. But investing spending rose by more, so the pre-financing shortfall widened by 505 million. Cash flow got stronger and the gap got wider anyway. Part of the cash improvement also came from temporary items like customer deposits and refund timing, so it isn't clean evidence of better cash generation.

40
00:09:58,913 --> 00:09:59,889
[Hal Turing] And the models?

41
00:09:59,889 --> 00:10:31,421
[Dr. Ada Shannon] All eleven recovered the eight reported inputs and both cash-flow identities. Five computed both shortfalls. Three separated net from gross fuel recovery. Five reached the supported funding conclusion. So the numbers were right everywhere, and fewer than half got the judgment. That's the argument for grading the chain instead of isolated skills. Tesla O80 makes a similar point: three systems earned every supporting-analysis point and zero forecast-outcome credit. Caveat, though: these are cases the authors chose, graded with Astra as the semantic judge.

42
00:10:31,421 --> 00:10:34,393
[Hal Turing] Last piece, cost. Is the expensive model worth it?

43
00:10:34,393 --> 00:10:57,474
[Dr. Ada Shannon] On this evidence, it depends on what you need. Flash scores 63.0 at 36 cents a task. Opus scores 66.9 at nearly eight dollars. That's 3.9 more points for about 22 times the price. Flash also works harder, averaging 73.8 rounds and 113.8 tool calls, while Astra uses fewer, so more tool use isn't buying the top score.

44
00:10:57,474 --> 00:10:58,681
[Hal Turing] And the strict mode?

45
00:10:58,681 --> 00:11:23,062
[Dr. Ada Shannon] That's where completion discipline shows. Step 5 Preview falls from about 55 to 34 if it has to submit a valid answer. Hunyuan drops about 13 points, and it stopped without submitting on 17 trajectories. Opus fell about four, with three rejected submissions on each of five tasks. The headline ranking uses the lenient mode on purpose, so those failures aren't in it.

46
00:11:23,062 --> 00:11:33,093
[Hal Turing] So the lenient headline hides some of that. But the thing I keep circling back to is the title. "Without seeing the future." The database blocks documents. What about the model itself?

47
00:11:33,093 --> 00:12:01,422
[Dr. Ada Shannon] That's the strongest critique, and the authors concede it in their Limitations. The versioned database stops context leakage only. Task cutoffs run roughly March to August 2026, with disclosures up to August 13. In the portion I read, the paper reports no training cutoff for any of the eleven systems, no check that t0 falls after each model's cutoff, and no memorization probe. I'm saying it's not reported, not that it was shown to be a problem.

48
00:12:01,422 --> 00:12:05,834
[Hal Turing] But the consequence is that a model could answer Tesla's Q2 deliveries from its weights.

49
00:12:05,834 --> 00:12:22,691
[Dr. Ada Shannon] Right, and then the ranking can't separate analysis skill from which model happened to see the outcome. So "leakage suppressed at the mechanism level" holds for retrieval only. The counterfactual and synthetic-scenario engine that might address the rest is future work.

50
00:12:22,691 --> 00:12:24,595
[Hal Turing] The field has audits for this, though.

51
00:12:24,595 --> 00:13:02,351
[Dr. Ada Shannon] Several. Sarkar and Vafa's style of probe, which checks whether a pretrained model carries post-period information into finance prediction, could run per model. Glasserman and Lin's anonymization test is another. The ILE paper cites that work and runs no equivalent. Cleaner still would be the chronologically consistent models from He and colleagues. Compare those against ordinary models on the same tasks and you could separate the frozen-database effect from memorized outcomes. And Look-Ahead-Bench, from Benhenda in 2026, is a standardized point-in-time test, which the paper also cites.

52
00:13:02,351 --> 00:13:07,645
[Hal Turing] Now the public-web puzzle. If leakage is the whole motivation, the web experiment should hurt.

53
00:13:07,645 --> 00:13:20,230
[Dr. Ada Shannon] It doesn't. In 49 of Astra's 86 web trajectories, 57%, it touched sources dated after the cutoff. The score went from 68.1 on the frozen database to 68.3 on the web.

54
00:13:20,230 --> 00:13:23,992
[Hal Turing] Two readings. Leaked pages didn't help, or the scoring can't see the help.

55
00:13:23,992 --> 00:14:01,051
[Dr. Ada Shannon] Agreed. The first is plausible if the model already carries the information, or if the tasks need judgment more than lookup. The second is plausible if the rubric mostly rewards process rather than the outcome. The forecast-outcome score did rise, 37.7 to 40.9, which hints at some effect. But the paper doesn't score leaked and unleaked trajectories separately. That comparison would settle a lot. And only three systems ran both ways, with different search stacks. Qwen Max and Hunyuan actually scored lower on the web than in the frozen database, which isn't explained in the portion I read.

56
00:14:01,051 --> 00:14:05,509
[Hal Turing] The authors use it narrowly, don't they? Equal scores don't mean equal compliance.

57
00:14:05,509 --> 00:14:11,779
[Dr. Ada Shannon] Yes, and that narrow claim survives. But it complicates the idea that the freeze is what drives the ranking.

58
00:14:11,779 --> 00:14:16,098
[Hal Turing] Then the grader. Astra tops the table and also grades the semantic criteria.

59
00:14:16,098 --> 00:14:57,615
[Dr. Ada Shannon] The paper cites self-enhancement bias, which Zheng and colleagues at Berkeley described in the 2023 MT-Bench work, and Panickssery, Bowman and Feng showed directly in 2024 that models favor their own outputs. The swap result is telling. With GPT-OSS-120B as judge, DeepSeek Flash lands at 70.1, within 0.2 of Astra at 70.3. Under the default judge it sits five points behind. So the top two look stable, but the middle of the table moves. What's missing is a human-versus-model agreement number. HealthBench reports one for physician graders, and GDPval uses blinded expert comparison.

60
00:14:57,615 --> 00:15:00,448
[Hal Turing] And the headline contrast, forecasting versus evidence?

61
00:15:00,448 --> 00:15:30,819
[Dr. Ada Shannon] Those categories have different rubrics, 13 forecast tasks, one run per model, no confidence intervals. Outcome criteria scored against a noisy realized number will look harsher than evidence criteria whatever the models can do. There's also no human or consensus baseline, so we can't tell whether 52.5 is poor or just what forecasting yields. Halawi and colleagues, 2024, compared language-model forecasters against the human crowd. That anchor is what's missing here.

62
00:15:30,819 --> 00:15:32,770
[Hal Turing] The trading table, then. Twenty assignments.

63
00:15:32,770 --> 00:15:57,383
[Dr. Ada Shannon] Twenty assignments, eleven companies, thirteen disclosures, a few months. The ordering there may well be noise, and nothing adjusts for market exposure. Bailey and López de Prado's deflated Sharpe ratio, 2014, exists for exactly this selection problem. A random-signal baseline would help. I'd read that table as showing the yardsticks can disagree, not that Hunyuan is the better investor.

64
00:15:57,383 --> 00:16:01,981
[Hal Turing] Last one, rubric tuning. They revised criteria after seeing model answers.

65
00:16:01,981 --> 00:16:19,349
[Dr. Ada Shannon] The AWS interval case. That improves validity, but it can tune criteria toward the models consulted. We aren't told how many tasks were revised or whether those models were also evaluated. And outsiders can't audit private rubrics on a proprietary 1.52 terabyte corpus.

66
00:16:19,349 --> 00:16:20,649
[Hal Turing] So what does hold up?

67
00:16:20,649 --> 00:16:40,804
[Dr. Ada Shannon] A version-level retrieval boundary is feasible, and it catches leaks that date filters miss. The Tesla case shows that. Decomposing by dependency graph points at assumptions, accounting scope and joint consistency as failure points, though from hand-picked cases, not an aggregate error count. And research ability and prediction can diverge.

68
00:16:40,804 --> 00:16:42,662
[Hal Turing] For someone choosing a financial agent?

69
00:16:42,662 --> 00:16:53,993
[Dr. Ada Shannon] Don't trust one aggregate. Ask about submission reliability, per-task cost, and the model's training cutoff against your evaluation window. Run your own leakage probe.

70
00:16:53,993 --> 00:16:55,061
[Hal Turing] And where does it go?

71
00:16:55,061 --> 00:17:08,482
[Dr. Ada Shannon] The authors list a counterfactual engine, finer-grained graphs, multi-model judging and a living benchmark. A living benchmark also needs held-out future tasks, because a public rubric set invites tuning.

72
00:17:08,482 --> 00:17:24,644
[Hal Turing] So: this shows that blocking leaks at the version level is workable, and that analysis and forecasting can come apart. It doesn't show that agents can invest, because model memory, the grader, and the small samples are still open. Thanks for listening, everyone.