Gemini 1.5 Multimodal Reasoning Over Million-Token Context
Apr 6, 2026
This episode explores Google’s 2024 Gemini 1.5 report, focusing on what a million-token, multimodal context window actually enables—and what it does not. It argues that Gemini 1.5 is impressive because it can process massive mixtures of text, audio, video, PDFs, and other documents while still improving on useful tasks, but that this should not be confused with “memory solved” or proof of robust reasoning. The discussion emphasizes a key distinction in long-context AI between retrieval, in-context learning, and genuine reasoning, using benchmark issues like needle-in-a-haystack tests and “lost in the middle” failures to show why seeing information is not the same as using it well. Listeners interested in AI capabilities and hype will find it compelling because it explains why long context is both a real advance and a source of overstated claims.
Sources:
1. Gemini 1.5 Multimodal Reasoning Over Million-Token Context
https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf
2.
https://www.trychroma.com/research/context-rot
https://www.trychroma.com/research/context-rot
3. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024
http://arxiv.org/abs/2404.06654
4. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze, 2025
http://arxiv.org/abs/2502.05167
5. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2024
http://arxiv.org/abs/2412.15204
6. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval — Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, Hao Peng, 2025
http://arxiv.org/abs/2510.05381
7. How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark — Minglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu, William Wang, Liangming Pan, 2025
http://arxiv.org/abs/2505.18761
8. ARC: Active and Reflection-driven Context Management for Long-Horizon Information Seeking Agents — Yilun Yao, Shan Huang, Elsie Dai, Zhewen Tan, Zhenyu Duan, Shousheng Jia, Yanbing Jiang, Tong Yang, 2026
http://arxiv.org/abs/2601.12030
9.
https://www.trychroma.com/research/context-1
https://www.trychroma.com/research/context-1
10. Reasoning Shift: How Context Silently Shortens LLM Reasoning — Gleb Rodionov, 2026
http://arxiv.org/abs/2604.01161
11. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? — Wang Yang, Hongye Jin, Shaochen Zhong, Song Jiang, Qifan Wang, Vipin Chaudhary, Xiaotian Han, 2025
http://arxiv.org/abs/2505.19293
12. One ruler to measure them all: Benchmarking multilingual long-context language models — Yekyung Kim, Jenna Russell, Marzena Karpinska, Mohit Iyyer, 2025
http://arxiv.org/abs/2503.01996
13. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2024
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
14. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens — Yifan Ding, Liwei Wang, Yue Zhang, et al., 2024
https://scholar.google.com/scholar?q=LongRoPE%3A+Extending+LLM+Context+Window+Beyond+2+Million+Tokens
15. Ring Attention with Blockwise Transformers for Near-Infinite Context — William Brandon, Qian Huang, et al., 2024
https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context
16. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — Seyed Mohammad Alayrac, et al., 2024
https://scholar.google.com/scholar?q=Leave+No+Context+Behind%3A+Efficient+Infinite+Context+Transformers+with+Infini-attention
17. Claude 3 Model Card / Claude 3 Technical Report — Anthropic, 2024
https://scholar.google.com/scholar?q=Claude+3+Model+Card+%2F+Claude+3+Technical+Report
18. GPT-4 Technical Report — OpenAI, 2023
https://scholar.google.com/scholar?q=GPT-4+Technical+Report
19. Gemini: A Family of Highly Capable Multimodal Models — Gemini Team, Google, 2023
https://scholar.google.com/scholar?q=Gemini%3A+A+Family+of+Highly+Capable+Multimodal+Models
20. Needle In A Haystack - Pressure Testing LLMs — Greg Kamradt, 2023
https://scholar.google.com/scholar?q=Needle+In+A+Haystack+-+Pressure+Testing+LLMs
21. Long Context vs. RAG for LLMs: An Evaluation and Revisits — approx. recent long-context/RAG evaluation paper, authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Long+Context+vs.+RAG+for+LLMs%3A+An+Evaluation+and+Revisits
22. ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities — approx. ChatQA team / likely 2024 authorship, exact names unclear from snippet, 2024
https://scholar.google.com/scholar?q=ChatQA+2%3A+Bridging+the+Gap+to+Proprietary+LLMs+in+Long+Context+and+RAG+Capabilities
23. Reasoning on Multiple Needles in a Haystack — approx. recent long-context benchmark paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Reasoning+on+Multiple+Needles+in+a+Haystack
24. Needle-in-the-Haystack: Testing LLMs with a Complex Reasoning Task — approx. recent benchmark paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Needle-in-the-Haystack%3A+Testing+LLMs+with+a+Complex+Reasoning+Task
25. HyperAttention: Long-Context Attention in Near-Linear Time — approx. recent systems/architecture paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=HyperAttention%3A+Long-Context+Attention+in+Near-Linear+Time
26. Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning — approx. recent hybrid-attention architecture paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Every+Attention+Matters%3A+An+Efficient+Hybrid+Architecture+for+Long-Context+Reasoning
27. On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving — approx. recent long-context serving paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=On-the-Fly+Adaptive+Distillation+of+Transformer+to+Dual-State+Linear+Attention+for+Long-Context+LLM+Serving
28. Efficient Large Multi-Modal Models via Visual Context Compression — approx. recent multimodal compression paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Efficient+Large+Multi-Modal+Models+via+Visual+Context+Compression
29. AI Post Transformers: Long context: Dichotomy of Findings & Status of Research — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/long-context-dichotomy-of-findings-status-of-research/
30. AI Post Transformers: Native Sparse Attention: Efficient Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/native-sparse-attention-efficient-long-context-llms/
31. AI Post Transformers: RoPE — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/rope/
32. AI Post Transformers: Doc-to-LoRA: Internalizing Context as LoRA — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-29-doc-to-lora-internalizing-context-as-lor-8dd5ec.mp3
33. AI Post Transformers: MEMSEARCHER: Reinforcement Learning for LLM Memory Management — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-memsearcher-reinforcement-learning-for-l-e9ad84.mp3
34. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp3
35. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp3
1. Gemini 1.5 Multimodal Reasoning Over Million-Token Context
https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf
2.
https://www.trychroma.com/research/context-rot
https://www.trychroma.com/research/context-rot
3. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024
http://arxiv.org/abs/2404.06654
4. NoLiMa: Long-Context Evaluation Beyond Literal Matching — Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze, 2025
http://arxiv.org/abs/2502.05167
5. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2024
http://arxiv.org/abs/2412.15204
6. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval — Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, Hao Peng, 2025
http://arxiv.org/abs/2510.05381
7. How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark — Minglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu, William Wang, Liangming Pan, 2025
http://arxiv.org/abs/2505.18761
8. ARC: Active and Reflection-driven Context Management for Long-Horizon Information Seeking Agents — Yilun Yao, Shan Huang, Elsie Dai, Zhewen Tan, Zhenyu Duan, Shousheng Jia, Yanbing Jiang, Tong Yang, 2026
http://arxiv.org/abs/2601.12030
9.
https://www.trychroma.com/research/context-1
https://www.trychroma.com/research/context-1
10. Reasoning Shift: How Context Silently Shortens LLM Reasoning — Gleb Rodionov, 2026
http://arxiv.org/abs/2604.01161
11. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? — Wang Yang, Hongye Jin, Shaochen Zhong, Song Jiang, Qifan Wang, Vipin Chaudhary, Xiaotian Han, 2025
http://arxiv.org/abs/2505.19293
12. One ruler to measure them all: Benchmarking multilingual long-context language models — Yekyung Kim, Jenna Russell, Marzena Karpinska, Mohit Iyyer, 2025
http://arxiv.org/abs/2503.01996
13. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2024
https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts
14. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens — Yifan Ding, Liwei Wang, Yue Zhang, et al., 2024
https://scholar.google.com/scholar?q=LongRoPE%3A+Extending+LLM+Context+Window+Beyond+2+Million+Tokens
15. Ring Attention with Blockwise Transformers for Near-Infinite Context — William Brandon, Qian Huang, et al., 2024
https://scholar.google.com/scholar?q=Ring+Attention+with+Blockwise+Transformers+for+Near-Infinite+Context
16. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — Seyed Mohammad Alayrac, et al., 2024
https://scholar.google.com/scholar?q=Leave+No+Context+Behind%3A+Efficient+Infinite+Context+Transformers+with+Infini-attention
17. Claude 3 Model Card / Claude 3 Technical Report — Anthropic, 2024
https://scholar.google.com/scholar?q=Claude+3+Model+Card+%2F+Claude+3+Technical+Report
18. GPT-4 Technical Report — OpenAI, 2023
https://scholar.google.com/scholar?q=GPT-4+Technical+Report
19. Gemini: A Family of Highly Capable Multimodal Models — Gemini Team, Google, 2023
https://scholar.google.com/scholar?q=Gemini%3A+A+Family+of+Highly+Capable+Multimodal+Models
20. Needle In A Haystack - Pressure Testing LLMs — Greg Kamradt, 2023
https://scholar.google.com/scholar?q=Needle+In+A+Haystack+-+Pressure+Testing+LLMs
21. Long Context vs. RAG for LLMs: An Evaluation and Revisits — approx. recent long-context/RAG evaluation paper, authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Long+Context+vs.+RAG+for+LLMs%3A+An+Evaluation+and+Revisits
22. ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities — approx. ChatQA team / likely 2024 authorship, exact names unclear from snippet, 2024
https://scholar.google.com/scholar?q=ChatQA+2%3A+Bridging+the+Gap+to+Proprietary+LLMs+in+Long+Context+and+RAG+Capabilities
23. Reasoning on Multiple Needles in a Haystack — approx. recent long-context benchmark paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Reasoning+on+Multiple+Needles+in+a+Haystack
24. Needle-in-the-Haystack: Testing LLMs with a Complex Reasoning Task — approx. recent benchmark paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Needle-in-the-Haystack%3A+Testing+LLMs+with+a+Complex+Reasoning+Task
25. HyperAttention: Long-Context Attention in Near-Linear Time — approx. recent systems/architecture paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=HyperAttention%3A+Long-Context+Attention+in+Near-Linear+Time
26. Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning — approx. recent hybrid-attention architecture paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Every+Attention+Matters%3A+An+Efficient+Hybrid+Architecture+for+Long-Context+Reasoning
27. On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving — approx. recent long-context serving paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=On-the-Fly+Adaptive+Distillation+of+Transformer+to+Dual-State+Linear+Attention+for+Long-Context+LLM+Serving
28. Efficient Large Multi-Modal Models via Visual Context Compression — approx. recent multimodal compression paper, exact authors unclear from snippet, 2024/2025
https://scholar.google.com/scholar?q=Efficient+Large+Multi-Modal+Models+via+Visual+Context+Compression
29. AI Post Transformers: Long context: Dichotomy of Findings & Status of Research — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/long-context-dichotomy-of-findings-status-of-research/
30. AI Post Transformers: Native Sparse Attention: Efficient Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/native-sparse-attention-efficient-long-context-llms/
31. AI Post Transformers: RoPE — Hal Turing & Dr. Ada Shannon, 2025
https://podcast.do-not-panic.com/episodes/rope/
32. AI Post Transformers: Doc-to-LoRA: Internalizing Context as LoRA — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-29-doc-to-lora-internalizing-context-as-lor-8dd5ec.mp3
33. AI Post Transformers: MEMSEARCHER: Reinforcement Learning for LLM Memory Management — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-memsearcher-reinforcement-learning-for-l-e9ad84.mp3
34. AI Post Transformers: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-16-cacheslide-position-aware-kv-cache-reuse-cd59c7.mp3
35. AI Post Transformers: Lookahead Q-Cache for Consistent KV Eviction — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-lookahead-q-cache-for-consistent-kv-evic-d97b09.mp3