Lookup vs. reasoning Illustrative
Needle-in-a-haystack tests lookup of one sentence. Toggle to see a schematic of how multi-fact reasoning is expected to decay. Colors are conceptual, not measured.
How a BABILong sample is made From the episode
Hover boxes for details.
Hide the facts in a book Illustrative layout
Task sentences are inserted in order among PG19 prose. Slot positions here are a schematic; real positions vary. The "Mary / milk" example is from the episode; QA1 and QA3 wording is a paraphrase.
QA1 → QA3: more facts to combine
Higher number, more supporting facts to find, in the right temporal order.
Effective context size Reported
Longest input where accuracy stays above 85%. Choose a task.
Many models fail before the haystack Reported
At 0K (no filler), 23 of 34 models reached 85% on at least one of QA1–QA3.
The 85% bar is a choice Illustrative
Schematic decay curve only. Drag the threshold: the crossing point moves. The paper never reran its analysis at 70% or 95%.
Three alternatives to a big window
Choose a method.
Reach beyond training length Reported
Log scale. Trained up to 32 segments (about 16K tokens).
Which headline fraction? Ranges as quoted
The episode notes the source gives different ranges and never reconciles them.
Specialists vs. generalists
Hover cells. Fine-tuned recurrent models were trained on this exact task.
Evidence ledger
What the paper shows, what the hosts infer, and what is unknown.
References
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack — Kuratov et al., 2024 (arXiv:2406.10149)
- Towards AI-Complete Question Answering (bAbI) — Weston et al., 2015
- Lost in the Middle — Liu et al., 2023
- RULER — Hsieh et al., 2024
- NoLiMa — Modarressi et al., 2025
- Recurrent Memory Transformer — Bulatov et al., 2022
- Scaling Transformer to 1M tokens and beyond with RMT — Bulatov et al., 2023
- Associative Recurrent Memory Transformer (ARMT) — Rodkin et al., 2024
- Infini-attention — Munkhdalai et al., 2024
- Gated Delta Networks — Yang, Kautz, Hatamizadeh, 2024
- Titans: Learning to Memorize at Test Time — Behrouz et al.
- Transformer-XL / Compressive Transformer (PG19) — Dai et al.; Rae et al.
- Counting-Stars and InfiniteBench — Song et al.; Zhang et al., 2024