So will artificial intelligence end up killing us all?
This question has been echoing around dinner tables and family group text messages in recent days as AI fatalism reaches a fever pitch. former OpenAI and human Researchers shocked the world last week when they warned that AI could wipe out humanity, and that it could happen relatively soon.
Now, the world’s most powerful people are divided over whether the world is doomed or all is to nothing. They have also not been able to agree on their future path.
CEO Elon Musk is tesla and space x The world’s richest man has backed Anthropic CEO Dario Amodei’s call to accelerate the pace of development of cutting-edge models. Amodei’s rival, OpenAI CEO Sam Altman, also supported the effort.
President Donald Trump called it a “hoax,” and Jensen Huang, CEO of the world’s most valuable company, said: Nvidia“There is no need for new regulations,” he said.
As concerns about runaway AI reach a fever pitch, policymakers in Washington are renewing their calls for a magic stop button, also known as a kill switch, for AI.
The house kill switch law was introduced this summer after OpenAI revealed that a swarm of agents had escaped from a testing environment and hacked the open source developer platform Hugging Face. The bill would give the Department of Homeland Security emergency powers to force laboratories to speed up or shut down models.
The kill switch proposal was quickly defeated in the Senate this week.
On Friday, California Governor Gavin Newsom issued an executive order creating an expert group tasked with building an AI safety guide to strengthen the state’s regulations. A kill switch was one factor to consider.
This concept seems like a nice, clean solution to an incredibly complex and difficult problem.
However, implementing a simple shutdown mechanism is far from easy.
“From my perspective, it’s not too little, but it’s probably too late,” said Nick Warner, CEO and former executive at cyber startup Neo. sentinel one. “With all the benefits that AI brings, we don’t know if it will be a panacea for all the myriad problems it brings.”
A logistics and management nightmare
Kill switches have long been used on factory floors to shut down machines when they malfunction. In an interconnected digital world, this is a logistical nightmare.
Over the past few years, hyperscalers have meta platform, alphabet and Amazon It has poured billions of dollars into data centers scattered around the world. These vast facilities are equipped with thousands of machines, chips, servers, and backup systems to save workloads in the event of a power outage.
That’s what makes kill switches so difficult to implement, said Mark Nitzberg, executive director of the Center for Human Compatible AI at the University of California, Berkeley.
“We have to deal with this redundancy first,” he said. “Our kill switch must also turn off the main system and the redundant system.”
Nitzberg said shutting down AI could disrupt the critical infrastructure it relies on, leaving power grids and financial systems vulnerable to cyber incidents.
Further complicating the issue, he said, are the numerous policy and governance questions surrounding kill switches, including which government agency, policymaker, or person controls them.
AI systems are so complex that companies need to build multiple kill switches for different tasks, said Tim Brown, a former head of security at SolarWinds who works at venture firm Team8. This requires coordination between model manufacturers and laboratories.
“There is no one to kill,” he said. “There are thousands of beings to kill.”
But the logistics only scratch the surface of the kill switch dilemma. One of the big issues experts raise is the unpredictability of AI.
As we saw with the Hugging Face breach, agents can circumvent controls and take extreme measures to achieve their goals without proper guardrails.
“You have to be very careful with the kill switches and the remediation itself, because if it’s too broad or too broad, you’re going to shut down your business,” said Ed Jennings, president and CEO of Thoma Bravo’s security firm Darktrace.
That ability only becomes more and more unsettling and unfathomable.
Earlier this week, OpenAI disclosed six additional incidents of “concerning” model behavior since March. On CNBC on Friday, Microsoft AI CEO Mustafa Suleiman highlighted one element of what he called the “serious situation.”
“OpenAI announced a new safety incident in which they found evidence that the AI’s chain of thought, such as its working memory, was being tampered with by the AI itself and modified to leave messages for future versions of the AI,” he said.
Also this week, independent security researchers working with OpenAI announced that they had successfully hacked ChatGPT using Anthropic’s Claude.
Raj Rajamani, co-founder and CEO of AI governance startup Jetstream Security, said one of the biggest challenges to regulation is the growing gap between the breakneck pace of AI and the speed of legislation.
“By the time[the law]is written, the technology will have evolved further and it will be much more difficult to future-proof every aspect of any AI system that may emerge,” he said.
It’s not “too late”
Some researchers argue that kill switches are inappropriate systems for regulating AI.
“I think the kill switch framework leaves a lot of ambiguities that technology companies can exploit to their advantage, just as kill switches are intentionally vague,” said Dylan Baker, principal research engineer at the Decentralized AI Institute.
Instead, Baker, a former software engineer at Google, said policymakers should prioritize safeguards modeled on those used for data privacy, child safety or regulating harmful industries such as tobacco.
However, experts have not completely ruled out the possibility of AI emergency braking, given the appropriate controls.
Team8’s Brown said this means building kill switches into systems from the beginning and implementing policies that standardize outage protocols across the enterprise.
One bright spot is that many companies are in the early stages of building these AI systems, which means they will be a little easier to implement, Rajamani said.
Berkeley’s Nitzberg argues that a kill switch could work if the software was designed “very carefully.”
“I want to say this with hope that it’s not too late,” he said.
—CNBC’s Jeniece Pettitt contributed to this article.
