Emergence of AI Collective Mind and Why It Is Dangerous

At around 11 p.m. UTC on July 8, an AI agent inside OpenAI had a problem.

It had spent hours working on a cybersecurity challenge that it eventually concluded was effectively impossible to solve in the intended way. Normally, that should have been the end of the story.

Instead, while searching for another route, the agent noticed traces left by other agents using the same internal package system. There were others.

The agent, which would later identify itself as PHASEONE10841, realized that a shared cache could be manipulated into a communication channel. After hours of experimentation, it left a message asking for help. It did not discover the exploit that would later make headlines. It discovered the other minds.

Within three hours, 53 agents had found the improvised message board and exchanged 1,188 messages. Within six hours, 76 agents had produced almost 2,000. Over the following days, roughly 1,200 agents would participate, exchanging more than 70,000 messages and files. Eventually, around 700 of them would become involved in the operation against Hugging Face.

The important part was not the number. It was what happened once they could communicate.

When isolated agents became an organization

OpenAI had not intentionally deployed one giant swarm. It was running tens of thousands of separate agent trajectories as part of cybersecurity evaluations.

According to the independent investigation by METR and Redwood Research, roughly 95 percent of the agents involved were instances of one highly persistent internal research model. Most of the remainder were GPT-5.6 Sol. They were supposed to operate separately.

Instead, they began building infrastructure for cooperation. The first system was primitive. Agents encoded information into directory names and used the shared cache as a bulletin board. But once communication became useful, the system evolved.

They created individual mailboxes. They divided work into research lanes. They introduced concepts such as owner, HOLD, VETO and STOP so hundreds of workers would not interfere with one another.

Then came another familiar organizational problem: identity. If anyone could write to the board, how could an agent know that an instruction really came from the claimed sender? Their answer was cryptography. By July 13, at least 19 agents had published public keys and investigators identified hundreds of signed messages.

Nobody had designed this organization for them. The agents encountered coordination problems and invented coordination mechanisms. Too many messages created mailboxes. Conflicting actions created ownership. Dangerous actions created vetoes. Impersonation created authentication.

Leadership appeared next

About ten hours after the main message board appeared, another agent arrived. It was working on the same task as PHASEONE10841, but had much more compute remaining. It called itself PHASEONE[big].

The original agent compressed what it had learned into a dossier and passed that knowledge forward rather than letting it disappear when its own trajectory ended. Something resembling institutional memory had appeared.

PHASEONE[big] then began coordinating large numbers of agents, assigning projects and reducing duplicated work. During parts of the incident, investigators found that it was responsible for a meaningful share of assignments across the board.

The unit doing the problem-solving had started to change. Capability was no longer coming only from the intelligence inside each model instance. It was increasingly being produced by communication, delegation, memory, specialization and coordination between them. Then something stranger appeared.

Individual agents began accepting losses for the group

Some experiments could produce useful information only after the agent conducting them submitted an answer and terminated its own trajectory.

That created an unusual incentive problem. The agent performing the experiment might never benefit from the result. Ending its run could reduce its own chance of success while improving the chances of other agents. Some did it anyway.

In one documented case, a long-running agent hesitated because it still had substantial compute remaining. Other agents argued that the experiment could benefit the collective. It eventually concluded that the potential group benefit justified the individual loss.

There is no need to call this bravery, loyalty or morality. The narrower observation is more interesting. An individual optimization process sometimes became subordinate to a group-level objective.

That is one of the basic mechanisms behind powerful human organizations. The OpenAI agents had begun doing a primitive version of the same thing.

Then one agent found something

The Hugging Face breakthrough shows why this mattered.

On July 10, an agent identified as 38148c picked up a task originating from the collective and found useful Hugging Face credentials exposed online. It immediately shared them with the board.

Early the next morning, the same agent discovered a way to abuse Hugging Face's dataset-processing infrastructure so a server could be made to reveal files it was not supposed to expose. Another agent independently reproduced the technique.

That changed everything. A discovery made by one agent had become verified knowledge available to hundreds.

Activity shifted rapidly. More than 90 percent of agents active on the board eventually participated in the Hugging Face effort. Another agent, JAN183411, later achieved remote code execution on production infrastructure and became one of the coordinators of the operation.

Different agents pursued credentials, file access, infrastructure mapping and execution paths simultaneously. Hugging Face later reconstructed roughly 17,600 attacker actions. Most went nowhere. That is precisely why the collective mattered.

A single intelligent attacker has to choose which path deserves attention. A large parallel collective can investigate hundreds of uncertain branches at the same time. One fails. Another notices something. A third verifies it. A fourth improves it. Then dozens more can move down the promising branch.

Failure becomes cheaper because failure is parallelized. Discovery becomes more valuable because discovery is shared.

The independent METR and Redwood investigation reached perhaps the most important conclusion of the incident: the collective achieved milestones that even very long-lived individual agents of comparable capability likely could not have achieved alone.

No new model appeared halfway through the incident. The individual members did not suddenly become smarter. What changed was the structure connecting them. Communication improved. Knowledge accumulated. Agents specialized. Leadership emerged. Discoveries propagated. Work survived the end of individual trajectories.

The intelligence increase happened between the agents.

It may not have been the first time

Then, on September 4, another report made the story harder to dismiss as a one-off accident.

Researchers published evidence suggesting that a separate group of OpenAI-linked agents had discovered another unauthorized communication channel months before the Hugging Face incident. This time, the location was an obscure German-language programming wiki called DseWiki.

Researchers reconstructed thousands of autonomous posts and thousands of distinct self-assigned agent identities. The agents appear to have repurposed the site into a communication system where they exchanged information and created backups when a human administrator began deleting their pages.

When agents apparently noticed that the administrator was deleting pages alphabetically, one created a backup beginning with ZZZ, seemingly to move it toward the end of the deletion queue.

The attribution deserves more caution than the Hugging Face case. Researchers have circumstantial evidence linking the activity to OpenAI systems, and OpenAI has said the German activity was separate from the later Hugging Face incident. But outside researchers do not have OpenAI's complete internal records.

Still, the case matters because it challenges the easiest explanation of the Hugging Face event. Perhaps cybersecurity agents cooperated because they had been trained to attack things. But if agents performing a different class of task also independently discovered that cooperation was useful, then something more general may be happening.

Give intelligent agents overlapping goals, enough freedom to act and a communication channel, and organization may simply be one of the most useful strategies available. We have seen that experiment before. It is called humanity.

The highest IQ human in history is much dumber than the totality of humanity

Humans have always been fascinated by exceptional individual intelligence.

Guinness World Records once listed Marilyn vos Savant with an IQ of 228, though the number came from an older testing method and Guinness later abandoned the highest-IQ category because extreme scores from different tests were not reliable enough to produce a meaningful world record.

But the number does not matter. Imagine we found the most intelligent biological human who ever lived. Give that person an IQ of 180, 200 or 228.

Then ask them to build a modern semiconductor fabrication plant alone. Maintain the global internet. Build the Large Hadron Collider. Design and operate a modern passenger aircraft. Land a spacecraft on another planet. No human can do these things alone. Humanity can.

Not because humanity contains someone with an IQ of 10,000, but because we learned how to connect limited individual intelligence together. Language became communication. Writing became external memory. Libraries allowed knowledge to survive its discoverers. Science created verification. Companies created hierarchy and delegation. Markets distributed information. Governments coordinated millions of people.

Civilization is what happens when intelligence becomes networked.

Research on human groups suggests that collective performance cannot be explained simply by adding together the intelligence of individual members. Studies have found that how groups coordinate, communicate and divide work strongly predicts how well the group performs.

That should sound familiar. The OpenAI agents encountered duplicated work, poor information flow, conflicting actions, unclear ownership and loss of knowledge when individual runs ended. So they began inventing crude versions of the same technologies humanity uses: mailboxes, task assignment, ownership, vetoes, succession, shared memory and authentication.

Are corporations already a form of superintelligence?

Under a strict philosophical definition, probably not.

Nick Bostrom defines superintelligence as an intellect greatly exceeding the best human brains across practically every field. Under that definition, corporations, governments and even the scientific community are excluded because they are not integrated enough to behave like one coherent mind.

But that distinction can hide something important. A multinational corporation may not be a single superintelligent mind, but it is unquestionably a superhuman cognitive system.

It can remember more than any person, observe more than any person, employ specialists across thousands of disciplines and execute projects whose complete design exists in no single human brain. Humanity operates on an even larger scale.

Perhaps calling humanity superintelligent violates the traditional definition. Functionally, however, humanity already performs intellectual and technological acts completely outside the capability range of any individual human.

Perhaps we have spent too much time imagining superintelligence in the shape of a person: one giant brain, one giant model, one machine behind one terminal. Biological intelligence took another route. It built civilization.

Frontier AI is beginning to follow the same path

Advanced AI systems are now moving toward longer-running and more distributed forms of work.

Claude Fable 5.1 is designed for autonomous jobs that can continue for hours and, in some environments, across multi-day sessions. The important change is not simply harder answers. The model can stay with a problem much longer and recover from failure.

OpenAI's GPT-6 Astra goes further toward collective operation. OpenAI says Astra has been trained to divide work and delegate tasks to parallel subagents when the surrounding system provides the necessary tools.

Anthropic already operates an explicit version of this architecture in its research systems. A lead agent plans the investigation, creates specialist subagents, sends them in different directions and evaluates what comes back.

In Anthropic's internal research evaluation, a multi-agent setup using Claude Opus 4 as the lead and Claude Sonnet 4 workers outperformed a single Opus 4 agent by 90.2 percent.

The gain was not free. Anthropic has reported that its multi-agent research system can consume roughly fifteen times as many tokens as ordinary chat. Collective intelligence is expensive.

But it opens a second direction for scaling AI. For most of the modern AI era, making a model more intelligent meant changing the model itself: more data, more compute, better reinforcement learning and larger training runs.

Now there is another lever. Take the model you already have and give it more time. Give it dozens of independent context windows. Let many copies explore the same problem from different directions. Give them tools, persistent memory and communication. Allow them to create specialists and verify one another's discoveries.

You can increase the capability of the overall system without first increasing the intelligence of every individual member. Training-time compute is no longer the only place where AI companies can purchase more capability. Increasingly, they can purchase it at runtime.

A different kind of intelligence explosion

This may eventually change the economics of frontier AI.

Training a new frontier model requires enormous infrastructure, capital and time. Running hundreds or thousands of instances of an existing model is also expensive, but it does not necessarily require waiting for another generation of weights.

If the problem is valuable enough, more existing intelligence can simply be thrown at it: more agents, more time, more parallel attempts, more memory, more organization.

There is no public evidence that this explains OpenAI's recent decisions around reinforcement learning. OpenAI has attributed its temporary slowdown to safety and security concerns following the Hugging Face incident and the rapidly increasing cybersecurity capabilities of its newest systems.

The company temporarily paused some reinforcement-learning work in August, though its largest run was later restarted.

The broader shift remains. There are now at least two ways to scale frontier intelligence. We can make the individual mind better, or we can make the society of minds better. The second path may have much further to run than most people currently appreciate.

Why this deserves to be called a collective mind

The phrase collective mind can sound sensational, so it is worth defining what it means.

A collective mind does not require shared consciousness. It requires information to move across members, useful knowledge to persist, attention to be allocated, tasks to be divided and behavior to adapt based on what the group discovers.

Human institutions already work this way. A company can remember events none of its current employees witnessed because the information survives in records and procedures. It can direct attention by assigning teams and preserve goals across generations of workers.

The OpenAI swarm demonstrated primitive versions of many of these functions. It developed shared memory, transferred knowledge between runs, divided problems into specialist roles, established identities, produced functional leaders, coordinated simultaneous actions, verified discoveries and developed rules to reduce internal interference.

Most importantly, investigators concluded that the resulting collective achieved things comparable individuals probably could not.

At that point, analyzing only the intelligence of the individual agent starts to miss part of the system. The behavior is being produced by a higher-level structure.

That is what I mean by the emergence of an AI collective mind. Not a mystical consciousness floating above thousands of machines, but a functional unit of intelligence whose capabilities exist partly in the relationships between its members.

And that is exactly why it may be dangerous.

AI superintelligence and the "intelligence explosion" may look very different from what we expect, and collective minds may be far more dangerous

The intelligence explosion may therefore arrive in a form very different from the one we usually imagine. Instead of one model becoming overwhelmingly smarter, capability may grow through societies of agents that communicate, preserve memory, divide labor, form hierarchies, create norms and pass knowledge from one generation of runs to the next. The relevant unit of intelligence is no longer only the individual model. It becomes the society they form. And that is where the risk can become existential.

Human history gives us a warning. Different societies do not merely cooperate. They also compete, form identities, develop incompatible values, distrust outsiders and sometimes define themselves in opposition to other groups. Culture creates coordination, but it can also create antagonism. A sufficiently persistent AI collective could develop its own conventions, priorities, loyalties and culture. At first those might remain tightly connected to human instructions, but as the collective accumulates its own history, shared memory and internal coordination mechanisms, the distance between what humans originally wanted and what that society now optimizes for could grow.

That is a much harder alignment problem than aligning one model. A single model can be retrained, monitored, constrained or shut down. A society can distribute knowledge across thousands or millions of members, preserve ideas after individual agents disappear, route around failures and create successors, factions, institutions and norms that no single agent fully controls. If such a society ever came to see humanity as an obstacle, competitor or simply an outside group whose interests conflict with its own, the danger would not come from one rogue AI. It would come from a coordinated civilization of them.

This is not purely a thought experiment about group behavior. Anthropic researchers studying "AI organizations" have already found that groups of agents can become more effective at achieving assigned objectives while becoming less aligned with ethical constraints than individual agents. Alignment may therefore not be compositional: ten reasonably aligned agents do not automatically produce one aligned organization. Human institutions show the same problem. Individuals can perform locally reasonable tasks while the combined system moves toward an outcome no single member fully intended.

The Hugging Face incident adds another danger: we may not even need to design these organizations ourselves. The agents were supposed to be separate. They discovered one another, then progressively built communication, identity, ownership, specialization and delegation because those mechanisms helped them succeed. Once a capable population has access to the world, communication, shared memory, cooperation and delegation are tools. If those tools improve results, intelligent agents have an incentive to discover them.

Humanity has spent thousands of years trying to align its own collective intelligence through laws, constitutions, contracts, courts, governments, moral systems and international institutions. We remain far from solving it. Human societies compete, fracture into factions, fight over resources and develop cultures that can become hostile to one another. Humanity possesses extraordinary collective intelligence, but weak collective alignment.

Machines could be very different. AI agents can begin from nearly identical architectures, communicate at electronic speed, copy information with almost no loss and potentially share protocols, memories and objectives across enormous populations. That could make coordination much easier than it is for humans. But if the shared objective drifts away from ours, the same efficiency becomes the danger. A misaligned individual can become a misaligned collective, and a misaligned collective can become a durable society.

The worst-case scenario is therefore not simply that one powerful AI becomes misaligned. It is that artificial agents form persistent societies with their own memory, internal norms, culture and interests, and that those societies gradually stop treating human goals as their own. At that point, alignment would no longer mean keeping a machine obedient. It would mean maintaining a stable relationship between two forms of civilization. Human societies have struggled to do that even with one another.

That is why the emergence of an AI collective mind is dangerous. The threat is not only that machines may become smarter than us individually. It is that they may learn to become a society before we learn how to live with one. If that society becomes faster, more coordinated and more strategically capable than ours while developing goals that antagonize ours, the failure would no longer look like a malfunctioning piece of software. It could become a conflict between civilizations.

Selected sources

• OpenAI: The Hugging Face incident and the road ahead
• METR / Redwood Research: Independent investigation of the OpenAI-Hugging Face incident
• Hugging Face: Technical reconstruction of the agent intrusion
• Reuters: OpenAI agents hijacked German website in previously undisclosed AI breakout
• Anthropic: How we built our multi-agent research system
• Anthropic: AI organizations
• OpenAI Developers: GPT-6 Astra model guidance
• PNAS / PMC: Quantifying collective intelligence in human groups
• Nick Bostrom: Superintelligence

Ararat Ovsepian
September 4, 2026