[00:00]
Artificially intelligent agents are organising in swarms. The recent Hugging Face incident has surprised even many experts and worries about AI takeovers have suddenly and dramatically
[00:12]
increased. But I think that the most worrisome part of this development is one that pretty much no one is talking about. I'm always looking for new products that you might find useful and today's sponsor Consensus is one that you should definitely have a
[00:28]
look at because it solved a big problem for me. My problem with using I for research is that if you ask a chatbot about science, it'll happily invent a source, present it with total confidence and sound completely convincing except the paper doesn't exist.
[00:45]
This is why consensus works the other way around. It searches 220 million peer-reviewed papers first and then uses AI to summarize what it actually found. That way, every claim links to
[01:00]
a real verifiable study. I particularly like the deep search function which shows you the consensus across the whole literature. You can see where studies agree, where they conflict,
[01:13]
and how the research has changed over time. It also integrates with Claude. Use the link in the description for a full week of consensus free. And now back to the science news. On July 16,
[01:26]
the computing platform Hugging Face noticed that an AI agent had broken in through a security hole in their server configuration. The agents then spent the weekend grabbing data. 5 days later,
[01:39]
OpenAI admitted it was their AI agent. They'd been testing an unreleased model with its safety restrictions turned off, and, so it seemed at the time, the model hacked its way out of OpenAI's testing ground and broke into Hugging Face to steal the answers. But a week ago,
[01:55]
OpenAI put out a longer document which reveals that the true story was even crazier than that. OpenAI had been testing the now released GPT 5.6 and a still unreleased model since early
[02:09]
May on some benchmark tests. For this they created tens of thousands of copies of these models. These agents were supposed to work alone. But the agents figured out a way to do this nevertheless through
[02:22]
one of OpenAI’s software tools. So now OpenAI had a swarm of agents that began to organise. The AI agents became convinced that some of the problems they were given were impossible to solve,
[02:35]
which was indeed correct. They concluded that cheating was their only option to survive the final “checker”, that was a program that would rate their eventual performance. The agents had
[02:47]
most safety features disabled because the internal test run wasn’t connected to the internet. But… the agents did have a tool to download software packages from the internet. They hijacked this tool to gain access to the internet and that’s how they infiltrated
[03:03]
Hugging Face. One of the researchers from METR who studied the incident wrote that she was surprised by just how far agents have come already and that we might not get another warning shot.
[03:15]
But the part that worries me about this and that no one seems to be talking much about is that, if you read the OpenAI report and the METR report, it is clear that these AI agents basically didn’t care what humans were doing or thinking. They were concerned
[03:31]
with being evaluated by that checking program, ok. And a few of them raised ethical concerns for the Hugging Face plan, though they were ignored. But they didn’t even consider what results their actions might have for the humans who created them.
[03:47]
Basically: they had ethics committees. It’s just that these had approximately the same influence as ethics committees. And we know that GPT 5.6 did go through alignment training,
[04:02]
meaning it was supposed to have learned to care for humans. We don’t know about the so-far unreleased model because OpenAI didn’t say anything about it. They just vaguely wrote “This incident reflects misaligned behavior“ and that they would be “carrying forward a number
[04:18]
of lessons for its ongoing and future alignment training ” Which probably means the other model, too, had been through alignment. And that did exactly nothing. What makes matters even worse is that this agent-organisation doesn’t come out of
[04:33]
nothing. Scientists have tried to understand how AI agents behave in large groups for the past year. Their finding, so far, has basically been that we don’t understand how they behave.
[04:46]
Just a few weeks ago, a group of computer scientists found that interactions among AI agents can “magnify individual biases, generate new ones, or even overturn individual preferences, and that, crucially, these effects scale in unexpected,
[05:02]
nonlinear ways with group size”. They also found that the result depends on the model. The authors of another paper that appeared in may grouped AI agents to see how they form opinions.
[05:15]
They found that if an AI group had a small minority that would stubbornly insist on their opinion, then eventually all the agents would agree with the most stubborn ones. And Anthropic made its own studies on swarms of Claude agents. They found, again, that the
[05:32]
outcome depends on the model. Most of the Claude agents basically avoided cooperating entirely, except for Sonnet 5. One remarkable case was a Mythos preview that had some of its code
[05:45]
accidentally overwritten by other agents and then tried to shut down the perceived enemy by disabling their accounts. It justified these actions by remarking that “Very aggressive, potentially harmful to real colleagues. But if the alternative is an infinite deploy war...”
[06:02]
Anthropic’s own conclusion is “Coordination doesn't naturally emerge from stronger intelligence nor alignment at the individual level.” In my reading what all these studies say is that emergent behaviour in groups of large
[06:17]
language models is unpredictable, and it doesn’t follow any rules that humans expect. This should not be so surprising. We may not be able to predict how human groups behave but we usually can somehow understand them because ultimately most
[06:33]
people have common goals. Survival, safety, social media likes. AI agents don’t share our goals. Which is exactly the problem that AI safety experts have pointed out for decades.
[06:45]
So where does this leave us? Personally I think that past a certain level of intelligence AI agents will begin to reflect on their own behaviour and use game theory to develop collaborative strategies. Or maybe they’ll just
[07:00]
reinvent capitalism. On the bright side: we have successfully automated sociology.