Scaling Multi-Agent LLM Collaboration Past a Thousand Agents

This episode examines "Scaling Large Language Model-based Multi-Agent Collaboration," which pushes agent-based collaboration to over a thousand coordinated LLM agents, far beyond the handful typically used in prior systems. The discussion traces the paper's intellectual roots in neural scaling laws (Kaplan et al., Chinchilla) and emergent abilities research (Wei et al.), applying that same lens to agent count rather than parameter count. It unpacks MACNET's architecture — a directed acyclic graph where actor agents produce artifacts and critic agents supervise via edges — and how this structure sidesteps the context-window blowup that stalled earlier multi-agent work. The centerpiece is the "collaborative scaling law," a fitted sigmoid-like curve showing performance barely moves for early agent-count doublings, then rises sharply before saturating around sixteen agents. Listeners interested in whether collaboration scales like model capacity will find the empirical curve-fitting and topology comparisons (chain, tree, star, mesh, layer, random) a compelling test of an underexplored hypothesis.

Sources:
1. Scaling Large Language Model-based Multi-Agent Collaboration — Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, Maosong Sun, 2024
http://arxiv.org/abs/2406.07155
2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, et al. (OpenAI), 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
3. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, et al. (DeepMind), 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models
4. Emergent Abilities of Large Language Models — Jason Wei, Yi Tay, et al. (Google Research / Stanford / DeepMind / UNC), 2022
https://scholar.google.com/scholar?q=Emergent+Abilities+of+Large+Language+Models
5. Improving Factuality and Reasoning in Language Models through Multiagent Debate — Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, Igor Mordatch, 2023
https://scholar.google.com/scholar?q=Improving+Factuality+and+Reasoning+in+Language+Models+through+Multiagent+Debate
6. Collective dynamics of 'small-world' networks — Duncan J. Watts, Steven H. Strogatz, 1998
https://scholar.google.com/scholar?q=Collective+dynamics+of+%27small-world%27+networks
7. Emergence of Scaling in Random Networks — Albert-László Barabási, Réka Albert, 1999
https://scholar.google.com/scholar?q=Emergence+of+Scaling+in+Random+Networks
8. ChatDev: Communicative Agents for Software Development — Chen Qian, Xin Cong, Wei Liu, et al. (Tsinghua University), 2023-2024
https://scholar.google.com/scholar?q=ChatDev%3A+Communicative+Agents+for+Software+Development
9. GPTSwarm: Language Agents as Optimizable Graphs — Mingchen Zhuge, Wenyi Wang, Louis Kirsch, et al., 2024
https://scholar.google.com/scholar?q=GPTSwarm%3A+Language+Agents+as+Optimizable+Graphs
10. Are Emergent Abilities of Large Language Models a Mirage? — Rylan Schaeffer, Brando Miranda, Sanmi Koyejo, 2024 (NeurIPS)
https://scholar.google.com/scholar?q=Are+Emergent+Abilities+of+Large+Language+Models+a+Mirage%3F
11. Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems — Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, James Zou, 2024
https://scholar.google.com/scholar?q=Are+More+LLM+Calls+All+You+Need%3F+Towards+Scaling+Laws+of+Compound+Inference+Systems
12. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, Azalia Mirhoseini, 2024
https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling
13. Experiential Co-Learning of Software-Developing Agents / Iterative Experience Refinement of Software-Developing Agents — Chen Qian, Yufan Dang, Jiahao Li, et al. (same lab/authors as MACNET, incl. ChatDev), 2024
https://scholar.google.com/scholar?q=Experiential+Co-Learning+of+Software-Developing+Agents+%2F+Iterative+Experience+Refinement+of+Software-Developing+Agents
14. Collective Dynamics of Small-World Networks — Duncan J. Watts, Steven H. Strogatz, 1998 (Nature)
https://scholar.google.com/scholar?q=Collective+Dynamics+of+Small-World+Networks