commentary
AI companies warn about the risk their models pose. Should it be on them to handle it?
Published
There are plenty of reasons to be skeptical of Silicon Valley’s warnings of A.I.-induced doom (Getty Images)
Imagine a stopped cable car sitting at the top of a steep hill in San Francisco. There’s a switch just below the car that would send the trolley onto one of two tracks. You don’t know where either track goes beyond the bottom of the hill, and you can’t see whether either track is obstructed. You’re also unsure of the tracks’ condition. The conductor is nowhere in sight. Do you release the brakes? And can anyone be mad at you if you do?
The answer is obvious. Of course you wouldn’t. Releasing a hurtling hunk of metal without knowing where it would go and what would happen is too likely to cause harm. The risk outweighs the reward of seeing the car go really fast.
But inside the brain trusts of Silicon Valley, billions of investor dollars and years of tortured rationalist arguments are clouding a similar argument about artificial intelligence. A.I. company CEOs, employee whistleblowers and industry boosters can’t stop crowing about the dangers posed by the products they’re rushing to create.
OpenAI CEO Sam Altman has worried that large-language models could be “more dangerous than nukes.” Speaking at a G20 summit in North Carolina earlier this month, he fretted about the disastrous effect A.I. agents could have on cybersecurity, warning that the rapid pace of large language model (LLM) development could cause things to go “very wrong very quickly.”
Altman is not alone in his fears. Here’s Anthropic, the A.I. corporation currently at war with the Pentagon and the Trump administration, in a blog post around about their Claude Mythos model:
“Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser. Given the rate of AI progress, it will not be long before such capabilities proliferate, potentially beyond actors who are committed to deploying them safely. The fallout—for economies, public safety, and national security—could be severe.“
Last weekend, OpenAI’s chief data scientist Jakub Pachocki voiced his concern in a blog post: “No one is prepared for the consequences of a continued rapid rise in machine intelligence.”
And just this week, Anthropic employee Jacob Coxon resigned from the company for fear that development of A.I. is “out of control.” These companies, he wrote on X, are “gambling with our lives” and building machines they “earnestly believe . . . could kill us all by the end of the decade.” Coxon, who previously worked at Anthropic’s competitor OpenAI, noted that OpenAI had not grappled with “the civilizational stakes.” Anthropic, he said, was actively ignoring them.
OpenAI data scientist Evan Hubinger backed Coxon’s story. “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” he wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
There are plenty of reasons to be skeptical of these Silicon Valley campfire stories. For one, the idea that Anthropic or OpenAI are capable of developing a model that could end all humanity is self-flattery.
There are plenty of reasons to be skeptical of these Silicon Valley campfire stories. For one, the idea that Anthropic or OpenAI are capable of developing a model that could end all humanity is self-flattery. For another, seeding the idea that they’re close to such breakthroughs can also have the advantageous effect of driving investment in an era where none of these companies have found a path to profitability. In other words: These fears could also be a way to boost their collective bottom lines.
The world-ending A.I. bogeyman becomes even less believable when the warnings are placed in proper context. Anthropic was quivering at the capabilities of their model in a blog post promoting their new security measures. Altman warned energy utilities about an A.I. attack on the grid this summer — while simultaneously pitching them on a suite of OpenAI products that might keep the lights on.
For the sake of argument, let’s take Altman and Anthropic co-founder Dario Amodei at their word. Their companies really are capable of creating a souped-up chatbot that could take the world back to the Stone Age. They could unleash a wave of A.I. agents that would cripple infrastructure, sow chaos and steal nuclear codes. But at what point should they be blamed for booting them up?
So far, A.I. boosters have been quick to take credit when their models hit a new benchmark in email-writing or 3D modeling. Social media platforms are littered with cheering sections gleefully anticipating mass unemployment in creative industries. The companies themselves relentlessly pitch A.I. integration to the moneymen at the top of service corporations, explaining that their bots will be able to cut out or reduce a pesky and expensive labor force. McClatchey, the company that owns a string of local and regional newspapers across the country, has recently come under fire from its own journalists for pushing a new generative A.I. tool to “summarize traditional articles and spit out different versions for different audiences.”
Want more sharp takes on politics? Sign up for our free newsletter, Standing Room Only, written by Amanda Marcotte, now also a weekly show on YouTube or wherever you get your podcasts.
The language used by the technology’s promoters shifts when discussion turns to cyberattacks, hacks and other crimes. A.I. goes from a suite of products its companies can sell to a force of nature. The anthropomorphizing runs wild as they share stories of supposed “AI civilizations” that make unanticipated decisions. They worry about a rogue superintelligence with misanthropic beliefs coming to kill us all. In this version of the story, A.I. stops being a tool and acts as a being that behaves in accordance with its own worldview.
All of this is hooey, of course. LLMs can’t have beliefs because LLMs don’t think. These models don’t make decisions; they carry out millions of probability calculations and pick the best one to answer the prompts they’re given. The models are lines of code written by people. They run on infrastructure that those people own. The tests are designed by the same people. If an A.I. model goes beyond the scope of a test or accidentally breaks free of its sandbox, you can’t blame the machine for its tester’s lack of guardrails. Failure to control A.I., then, points to a lack of imagination from the creator, not from the genius of the machine.
The GOP is all-in on A.I., and they have created a handy metric for understanding where to lay the blame when a machine’s output causes harm. You’ve heard it at nearly every Republican rally for decades: Guns don’t kill people, people kill people. The same holds true for any hypothetically murderous A.I. models.
CEOs can’t take credit for the successes of A.I. while avoiding the negative externalities. They’re both the results of real decisions made by actual thinking people. The blame for any harm the technology inflicts should lay with its creators.
If the logic I used from Second Amendment maximalists makes you uncomfortable, let’s return to the trolley problem. If Amodei loosed a 15,000 pound vehicle that smashed through the lobby of the Salesforce building and maimed several future stars of the software-as-a-service sector, we wouldn’t be looking to lock up the cable car. We would be trudging up the hill to find the man who threw the switch.
