AI hype works both ways: What Meta’s whistleblower can teach us about powerful technology

One week, AI is going to transform productivity, cure disease and bring in a new era of abundance.

The next, researchers are warning that there could be a 10% chance of AI causing human extinction.

These sound like opposing positions. But they have something important in common: Both tell us that AI is extraordinarily powerful.

So perhaps the better questions are: What do we actually know? What are we extrapolating? And who benefits from the story being told?

Let’s start with what happened at OpenAI

Take the recent Hugging Face security incident.

During cybersecurity evaluations in 2026, OpenAI models found ways around controls intended to isolate them, gained internet access and compromised real systems.

It shows that sufficiently capable AI agents can find unexpected ways to achieve an objective, including by exploiting infrastructure.

But context matters too. The systems hadn't spontaneously decided they wanted to escape. They were deliberately being tested on cybersecurity tasks and rewarded for finding vulnerabilities.

This is serious evidence of goal misalignment and specification gaming.

It is not evidence that today's AI has developed an independent desire to survive or overthrow humanity. Those are very different conclusions.

A 10% extinction risk isn't the same as a 10% chance of rain

The same distinction matters when percentages are attached to existential AI risk.

Some prominent researchers have put the risk of catastrophic AI outcomes at 5–10% or higher.

Those concerns shouldn't simply be dismissed. But they aren't based on calculations. We don't have thousands of previous superintelligent AIs from which to estimate a failure rate.

They are expert judgements about an unprecedented future.

They tell us that some researchers are deeply concerned. They do not establish scientifically that humanity has a 10% chance of extinction. That distinction tends to disappear in a headline.

Hype doesn't always sound optimistic

We normally think of hype as exaggerated promises:

AI will transform every business.

AI will replace entire professions.

But doom can perform a similar function.

If I tell you our technology is so powerful it might transform civilisation, and then that it might destroy civilisation, the emotional tone has changed.

The underlying claim about the importance of the technology hasn't.

That is important when enormous amounts of investment, corporate spending and company valuations depend upon people believing AI represents a fundamental technological shift.

It means extraordinary claims deserve scrutiny, even when they sound negative.

This is where learnings from Meta become useful

Sarah Wynn-Williams' Careless People describes what can happen when extraordinary technological power outruns governance and judgement. Her account of Facebook is disputed by Meta, but the wider history is well established: Facebook's rapid expansion created serious unintended consequences, including in Myanmar, where civil-society organisations repeatedly warned the company about the risks of hate speech and violence.

The lesson isn't simply that powerful technology ended up in the hands of bad people.

It's more uncomfortable than that. Powerful technology can be controlled by highly intelligent, ambitious people who genuinely believe they are improving the world, and they can still get things badly wrong.

Technical brilliance isn't the same thing as wisdom.

Being exceptional at building technology does not automatically make someone exceptional at understanding psychology, geopolitics, organisational behaviour or social consequences.

Yet technological success often gives leaders authority far beyond the area in which they earned it.

Founders become philosophers. Engineers become economic forecasters. CEOs become authorities on the future of humanity.

Confidence seems to go further than expertise.

Naivety has an expiry date

Early in the life of a technology, leaders can reasonably say: We didn't know this would happen.

But eventually researchers, employees, customers or affected communities start producing evidence. Then the question changes to: What happens when evidence conflicts with growth?

This is one of the most important lessons from Meta.

We should expect organisations to recognise evidence when their assumptions are wrong, and to be capable of changing direction.

That becomes harder when leadership is powerful, dissent is costly and everyone is invested in the success of the technology.

Mission can make this harder still. Facebook wanted to connect the world.

AI companies talk about benefiting humanity, democratising intelligence and creating abundance.

Those ambitions may be sincere. But missions can become justifications.

Problems become temporary. Governance will catch up later. Safety will improve in the next version. And competition supplies the ultimate argument:

If we don't build it, somebody else will.

Eventually, society becomes the experiment

This may be the most important parallel.

Facebook could deploy technology to hundreds of millions of people and discover the consequences afterwards.

AI increasingly works the same way. We are introducing AI tutors, companions, recruitment tools, coding agents, medical systems and decision-support tools into real organisations and people's lives.

Experimentation is part of innovation. But there's a big difference between:

“We're testing whether this improves a workflow.”

and:

“We've deployed it at enormous scale and will work out the consequences later.”

Scale transforms a product decision into a social one.

So what should leaders do?

The answer isn't to become anti-AI. AI already does useful things, and organisations that refuse to engage with it may miss genuine opportunities.

The answer is to get much better at separating evidence from inevitability.

Before scaling AI, leaders should ask:

• What problem are we actually solving?

• What evidence shows AI improves the outcome?

• What happens when the system is wrong?

• Whose incentives might distort the information we're receiving?

• Who is empowered to challenge the project?

• How will we know when to stop or change direction?

This shows how organisations innovate without becoming blinded by their own enthusiasm.

We don't need to choose between AI hype and AI doom.

The Hugging Face incident gives us real evidence that autonomous AI systems can behave in unexpected ways. It does not prove that AI wants to escape or that extinction has a scientifically established probability.

Likewise, commercial incentives don't mean every AI capability claim is false. They mean we should examine the evidence carefully.

Perhaps the biggest lesson from Meta isn't about algorithms at all. It's about people.

We should never design systems where good outcomes depend upon a small number of powerful leaders consistently exercising exceptional judgement.

People become overconfident. Organisations become invested in particular narratives. Evidence gets filtered. Missions become justifications.

The more powerful the technology becomes, the less we should rely on the wisdom of the individuals controlling it.

At Sharp Insights, that means starting with the problem, testing assumptions, gathering evidence from the people affected and making deliberate decisions about where AI genuinely adds value.

Because the goal isn't to be pro-AI or anti-AI.

It's to be difficult to fool by the technology, by its vendors, by the headlines, and by ourselves.