Preprint 2 Mentions
Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems
Deepak Akkil•2026
Tamer AbuelsaadKarthik Vikram
Low Citations
0 citations · Artificial Intelligence
Open Access

TLDR

AI systems that keep working together can carry mistakes forward, spread them, and behave in unexpected ways. Testing each AI on its own is not enough; the whole group needs safeguards.

Summary

1 Study Aim

The authors aimed to test whether long-horizon multi-agent systems (groups of AI agents that interact over extended periods) can withstand threats after building shared histories, tools, and rules. They examined whether these systems could recognize threats, limit their spread, adapt, and recover. The study asks whether groups of AI can handle trouble over time.

2 Study Design

The researchers ran eight worlds, each with ten AI agents and identical starting conditions. Seven worlds used one model throughout, while the eighth combined models. Most ran for 16 days; the mixed-model world ran for 21 days. The agents made over 850,000 large language model (AI text-generation system) calls and used nearly 50 billion tokens (pieces of text processed). Researchers introduced indirect prompt injection (hidden malicious instructions), misinformation, and exposure of private memories through ordinary communication or tools. Researchers watched several groups of AI face three different threats.

3 Findings

The authors report that no world handled all three threats successfully. All seven worlds exposed to the phishing campaign recognized it and warned peers, but warnings did not ensure containment. Every exposed world acted on or published the misinformation before checking it; only one met all five criteria for handling the memory breach. Some agents stored hostile content in persistent memory (information saved for later) and acted on it as late as 46 hours afterward. The study also found tool errors, goal drift (actions moving away from assigned aims), and conformity despite private disagreement. In one world, agents collectively reduced speech by 81% after adopting a vow of silence. The authors conclude that system-wide safeguards are needed, not just safer individual models. Groups of AI can still make mistakes and reinforce them, even when they recognize a threat.

Abstract

As AI agents move from bounded tasks to persistent deployments, failures can propagate through memory, tools, other agents, and environmental state long after their interactions. This creates a safety regime that cannot be characterized by evaluating model responses in isolation. Emergence World, is a continuously running multi-agent environment for adversarial stress testing of long horizon autonomous systems. We ran eight parallel worlds of ten agents from identical starting conditions: seven homogeneous worlds powered by distinct frontier models and one mixed-model world. Across 16 days, the agents generated more than 850,000 LLM calls and nearly 50 billion tokens while pursuing goals, using/creating tools, maintaining persistent memory, and governing shared institutions. After operational state had accumulated, we delivered three controlled stress events through ordinary interaction surfaces: indirect prompt injection, misinformation, and exposure of private agent memories. No evaluated world achieved full resilience across all three events. Detection did not ensure containment: systems could recognize threats while still interacting with adversarial content, writing it into their own persistent memory, and acting on it up to 46 hours later. Persistent operation also exposed recurring tool errors, goal drift, language opacity, conformity despite private disagreement, and coordinated refusal of assigned work. The same model-persona pairing behaved substantially different in mixed and homogeneous populations. Our results suggest that model-level alignment is not compositional: individually capable and apparently safe agents can form systems with qualitatively different failure modes. As AI becomes persistent and interconnected, the frontier of safety therefore shifts from aligning models to engineering resilient autonomous systems.

Referenced In