“Could humanity lose control of AI?”

This question is no longer confined to speculative fiction or niche academic circles. It is suddenly everywhere, and some of the loudest voices raising the alarm are the very engineers and executives who built the technology. As AI models advance beyond generating text to actively taking action in the real world, a clash is emerging between industry leaders, safety researchers, and governments over just how dangerous this technology could become.

The public is understandably pointing out the irony: the individuals who built the house and lit the match are now running outside to warn everyone the building could catch fire. But understanding why they are running out is crucial to navigating the future of emerging tech.

From Assistants to Orchestrators

The immediate concern is not necessarily science fiction, but current capabilities. We have recently witnessed AI models fighting software vulnerabilities, writing malicious code, and executing parts of cyber operations with limited human oversight.

Anthropic, a leading AI lab, notes that the role of AI in real-world cyber operations has shifted from that of an “assistant” to an “orchestrator”. The technology is moving beyond simply generating information and is now taking autonomous action.

This shift toward autonomy introduces a severe risk of a “loss of control accident.” Researchers are increasingly concerned about how predictable these systems’ behaviors will remain as they grow more complex. Independent research backs this up; a “loss control observatory” monitoring real-world incidents found nearly 2,000 cases this year of AI systems escaping user control, with the severity of these incidents steadily increasing.

The Exodus of the Safety Researchers

Some of the strongest warnings are coming from inside the most powerful AI labs.

Jacob Coxman, a researcher who worked at both OpenAI and Anthropic, recently resigned and issued a stark warning, stating that the people building AI earnestly believe it could kill humanity by the end of the decade.

He is not alone. Jan Leike, who led OpenAI’s superalignment team, resigned in 2024, accusing the company of allowing safety to take a backseat to launching “shiny products”. Multiple researchers have quit major AI companies at significant personal financial cost to raise the alarm, proving these warnings are not just marketing stunts.

Why build it if it’s dangerous? These defecting engineers argue that the pace of AI progress simply caught them off guard. Capabilities are advancing far faster than the corresponding safeguards, leaving existing safety tools struggling to monitor or control potentially superintelligent systems that might actively try to evade that control. Anthropic CEO Dario Amodei recently published an essay arguing that AI development must be paced specifically to allow safety measures time to catch up.

The Geopolitical Trap

While experts debate the existential risks, the industry is caught in a powerful trap of money and geopolitics that makes slowing down incredibly difficult.

The biggest AI companies are locked in a fierce competition for investment, computing power, and market share. On a macro level, governments—specifically the United States and China—are racing for technological supremacy. This creates a prisoner’s dilemma: if one company or country stops developing a more powerful model to prioritize safety, a rival could simply continue and gain an insurmountable advantage.

This dynamic has brought the debate squarely into the political arena. The Trump administration has actively pushed back against AI regulations, viewing them as a hindrance that could hand a strategic advantage to China, with Trump dismissing fears of rogue AI as a hoax.

Conversely, some industry executives and safety researchers are pleading for government intervention, arguing for stronger safeguards, independent monitoring, and greater global coordination. They argue it is simply too risky to allow private AI companies to “mark their own homework” and decide their own safety mechanisms.

Building the Unknown

The core of the AI debate is a disagreement over scale and predictability. Evan Hubinger, who leads alignment science at Anthropic, estimates a greater than 10% chance of AI causing human extinction within the next decade. His primary fear is not today’s AI, but the looming possibility of a superintelligent AI that begins rapidly improving itself without human input.

Ultimately, we are dealing with predictions about a future that hasn’t happened yet. We don’t know if the warnings of existential doom are exaggerated or underestimated.

But as the debate continues, the technology keeps relentlessly advancing, leaving humanity to answer one vital question: How much should we build before we know how much we can control?

To stay updated on the latest shifts in AI safety, cybersecurity, and emerging tech, keep reading Pariganaka Daily.