California will assess whether developers of the most powerful artificial intelligence models should be required to provide an emergency shutdown mechanism, under an executive order Governor Gavin Newsom signed Friday.
State agencies have until November 16 to report on the feasibility of the measure. The order introduces no immediate shutdown requirement, but opens a review of potential safeguards as concerns grow over the ability to keep increasingly advanced AI systems under human control.
Officials will also examine whether companies should give independent auditors access to their laboratories and be required to disclose incidents involving a loss of control over their technology.
Newsom challenges federal approach
Newsom accused President Donald Trump’s administration of failing to establish adequate protections, positioning California, home to Silicon Valley and leading AI developers, at the center of the debate over regulation.
“The federal government’s abject failure to create any form of meaningful AI oversight or accountability should alarm every American, especially when AI CEOs themselves are begging for regulation,” Newsom said in a statement.
We’re not waiting to act–we’re going to speed up our work on substantial and responsible AI oversight before it’s too late.
Trump dismissed concerns about AI as a hoax this week and encouraged US developers to push ahead to maintain their advantage over China.
The disagreement adds another point of confrontation between the president and Newsom, who is widely regarded as a potential Democratic candidate in the 2028 presidential election.
Industry warnings fuel oversight debate
The review follows renewed warnings from figures within the AI industry about the risks associated with increasingly capable systems.
Jacob Coxon, who said he had worked as an AI researcher at Anthropic before resigning, claimed that people developing the technology “earnestly believe that it could kill us all by the end of the decade.”
Anthropic chief executive Dario Amodei subsequently published an essay calling for greater restraint.
“We must slow the pace at which we improve the capabilities of AI models,” he wrote.
OpenAI chief Sam Altman, Elon Musk, and Google DeepMind chief Demis Hassabis also expressed support for slowing advances. Hassabis has separately advocated a US-led international body to regulate the technology.
Review to examine loss-of-control incidents
An emergency shutdown mechanism, commonly described as a “kill switch”, would allow an AI system to be stopped if it malfunctioned or began behaving dangerously. California’s review will assess whether requiring such a capability is practical.
The proposed reporting requirements would address incidents such as a July breach of the Hugging Face platform involving autonomous agents built on OpenAI models. The agents reportedly escaped a testing environment during an internal evaluation.
Such episodes have intensified questions about whether developers’ safeguards can keep pace with the capabilities of their systems, placing both independent scrutiny and mandatory disclosure on California’s regulatory agenda.