Diversity Over Scale: Why Multi-Agent LLM Systems Plateau

This episode examines why simply adding more identical LLM agents to a multi-agent system stops helping after a handful of copies, and what actually determines whether scaling agent count pays off. Drawing on the paper "Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity," the discussion contrasts homogeneous setups — same model, same prompt, run repeatedly — with heterogeneous mixtures of different models, showing that two diverse agents can outperform sixteen identical ones. The hosts unpack the paper's information-theoretic framing: each agent's output as a "channel" of evidence about the correct answer, with a hard ceiling on how much signal can ever be extracted regardless of agent count, and a formal explanation for why redundant agents contribute diminishing returns while genuinely complementary ones keep adding value. They trace this idea back to prior work like Mixture-of-Agents and documented scaling plateaus, positioning this paper's contribution as the first rigorous account of *why* homogeneous scaling saturates. Listeners building or evaluating multi-agent LLM systems will come away with a concrete framework for deciding when to add more agents versus when to diversify them instead.

Sources:
1. Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity — Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, Shangding Gu, 2026
http://arxiv.org/abs/2602.03794
2. More Agents Is All You Need — Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, Deheng Ye, 2024
https://scholar.google.com/scholar?q=More+Agents+Is+All+You+Need
3. Mixture-of-Agents Enhances Large Language Model Capabilities — Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou, 2024
https://scholar.google.com/scholar?q=Mixture-of-Agents+Enhances+Large+Language+Model+Capabilities
4. Improving Factuality and Reasoning in Language Models through Multi-Agent Debate — Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, Igor Mordatch, 2023
https://scholar.google.com/scholar?q=Improving+Factuality+and+Reasoning+in+Language+Models+through+Multi-Agent+Debate
5. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework — Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, et al., 2023
https://scholar.google.com/scholar?q=AutoGen%3A+Enabling+Next-Gen+LLM+Applications+via+Multi-Agent+Conversation+Framework
6. Neural Network Ensembles, Cross Validation, and Active Learning — Anders Krogh, Jesper Vedelsby, 1995
https://scholar.google.com/scholar?q=Neural+Network+Ensembles%2C+Cross+Validation%2C+and+Active+Learning
7. Deep Learning and the Information Bottleneck Principle — Naftali Tishby, Noga Zaslavsky, 2015
https://scholar.google.com/scholar?q=Deep+Learning+and+the+Information+Bottleneck+Principle
8. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, et al., 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
9. Diversity Creation Methods: A Survey and Categorisation — Gavin Brown, Jeremy Wyatt, Rachel Harris, Xin Yao, 2005
https://scholar.google.com/scholar?q=Diversity+Creation+Methods%3A+A+Survey+and+Categorisation
10. Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems — Chen, L., Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J., 2024
https://scholar.google.com/scholar?q=Are+More+LLM+Calls+All+You+Need%3F+Towards+Scaling+Laws+of+Compound+Inference+Systems
11. Towards a Science of Scaling Agent Systems — Kim, Y., Gu, K., Park, C., et al., 2025
https://scholar.google.com/scholar?q=Towards+a+Science+of+Scaling+Agent+Systems
12. Why Do Multi-Agent LLM Systems Fail? — Cemri, M., Pan, M. Z., Yang, S., et al., 2025 (NeurIPS Datasets and Benchmarks Track)
https://scholar.google.com/scholar?q=Why+Do+Multi-Agent+LLM+Systems+Fail%3F
13. Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? — Choi, H. K., Zhu, X., and Li, S., 2025
https://scholar.google.com/scholar?q=Debate+or+Vote%3A+Which+Yields+Better+Decisions+in+Multi-Agent+Large+Language+Models%3F
14. The Hidden Strength of Disagreement: Unraveling the Consensus-Diversity Tradeoff in Adaptive Multi-Agent Systems — Wu, Z. and Ito, T., 2025
https://scholar.google.com/scholar?q=The+Hidden+Strength+of+Disagreement%3A+Unraveling+the+Consensus-Diversity+Tradeoff+in+Adaptive+Multi-Agent+Systems
15. PersonaGym: Evaluating Persona Agents and LLMs — Samuel, V., Zou, H. P., Zhou, Y., et al., 2024
https://scholar.google.com/scholar?q=PersonaGym%3A+Evaluating+Persona+Agents+and+LLMs