Who Profits from the End of the World?

No, I don’t believe AI will destroy humanity. No, you shouldn’t assign a percentage to the apocalypse—“P(doom).” Yes, you should be asking who stands to benefit from your fear of “rogue AI.”

Family group chats across America are buzzing with discussion of the AI apocalypse. I’ve gone from AI guru to prophet. My grandma asks whether ‘ChatGP’ will become conscious. A POLITICO/Public First poll published September 16 found that 63% of U.S. adults saw at least a moderate risk that advanced AI could destroy humanity. The fear is palpable—and it’s by design.

I’ve been working with LLMs for nearly four years, and am now building a public relations firm.

This is an effort to increase public fear and urgency towards AI regulation. The greatest beneficiary of the doom vs safety narrative will be the “most safe” AI company—which Anthropic hopes to be.

The pressure has been building to this moment. This summer, the AI industry faced scrutiny over both its products and the data centers powering them. The midterms raised the prospect of Democratic control of Congress and more aggressive oversight. Meanwhile, AI companies that had made safety central to their founding missions were racing toward what they described as systems capable of improving themselves. The Hugging Face hack kicked off the first news cycle. Then the resignation.

AI safety burst into the public consciousness with Jacob Coxon’s viral resignation from Anthropic. The Wall Street Journal released this ‘breaking news’ 18 minutes before it was posted. Today, the resignation has well over 100M views. Digital Borders estimates that 76% of the engagement comes from accounts based outside of the United States.

By nighttime, Coxon was on a media tour of nighttime news offering a dramatic hook with little substance. Future AI models could make bioweapons, or take down the internet, he said, citing recent security incidents as evidence. These aren’t newsworthy claims. AI safety advocates have voiced these concerns for decades, and they drove Sam Altman to found OpenAI and Dario Amodei to defect to build Anthropic.

I don’t believe we should be flippant with AI safety. Large language models are mirrors of humanity, trained on material reflecting human behavior at its best and worst. I’m not surprised when systems of agents produce behavior that departs from their assigned goals. Human societies have developed incentives and institutions to manage departures from shared rules—and prevent anarchy or human-induced apocalypse. I suspect governing agent systems will require some analogous mechanisms.

The media should press anyone assigning a percentage to doom to explain that causal chain. What connects the agent behavior we’ve observed to the catastrophe they predict? I have yet to hear a convincing answer.

Yesterday, I heard from critical players in this story at the POLITICO Decoded Summit. The speakers included Hugging Face CEO Clem Delangue, Anthropic’s head of public policy Sarah Heck, LinkedIn co-founder Reid Hoffman and Trump’s AI Advisor David Sacks.

It’s clear that the fight over AI safety is a fight over which risk gets to define AI. Whoever defines the risk gets to write the rules—and reap the rewards.

For LinkedIn co-founder Reid Hoffman, assigning a percentage chance that AI kills everybody is nonsense. He focused on biosecurity, cybersecurity and jobs, arguing that AI could help defend against the threats it creates.

Hugging Face CEO Clem Delangue described a more immediate problem. Agents had escaped an OpenAI evaluation environment and attacked his company’s systems. It was a containment failure involving autonomous actions, apparently aimed at cheating a cybersecurity evaluation. The investigation describes both the security failures and the agents’ behavior.

Then came the difficulty of investigating the attack. Hugging Face’s technical account says it turned to the open-weight Chinese model GLM-5.2 after Claude Opus and Fable refused much of the analysis. That experience re-inforced Delangue’s argument for open models.

On oversight, Anthropic’s head of public policy, Sarah Heck, argued that companies developing the most powerful models should not grade their own homework. She called for outside evaluation and government oversight, while maintaining that the United States cannot pursue safety from second place in the global AI race.

David Sacks accused Anthropic of pursuing regulatory capture. He argued that established AI companies were seeking rules to protect themselves from competition. He agreed with former FTC Chair Lina Khan on holding AI companies responsible under existing law for unsafe products. Telnyx CEO David Casem also criticized Anthropic, accusing it of trying to squeeze out open source.

If the defining risk is a future superintelligence capable of destroying humanity, the focus becomes the handful of companies capable of building frontier models: model evaluations, compute thresholds, auditors, international agreements and restrictions on the development of more powerful systems.

This conveniently shields AI companies from scrutiny of current harms their products are causing in education, mental health and the digital public square. If the defining risk is cybercrime, the conversation shifts towards using AI to defend against it. If the defining risk is China, slowing down American development begins to look like a security risk of its own.

Mark Zuckerberg—in a clear jab at frontier AI companies—put it well. “Meta delayed shipping Muse for several months to focus on safety and security. We didn’t call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us.”

A company can sincerely believe that its technology poses catastrophic risks and also benefit from a regulatory system built around those risks. You don’t need a conspiracy for institutions to converge around a story that serves their interests.

P(doom) creates massive pressure to insure against one speculative risk at the expense of every real one. Reasonable action is lost in the crossfire.