California Proposes Emergency Powers to Shut Down Dangerous AI Systems

Governor Gavin Newsom at an AI workforce event, August 7, 2025, archive photo
URL has been copied successfully!

Governor Gavin Newsom at an AI workforce event on August 7, 2025. Archive photo: Office of the Governor of California / Wikimedia Commons, public domain; cropped.

California proposed giving the state authority to order emergency shutdowns of dangerous AI systems Wednesday, releasing recommendations that would also require stronger independent oversight and faster reporting of serious safety incidents.

The recommendations are a blueprint for legislation, not newly enacted shutdown powers. Governor Gavin Newsom’s November 18 announcement presented the proposals as the next step in strengthening California’s existing AI safety framework.

The Government Operations Agency report, dated November 16, recommends that the state be able to intervene when an AI model poses an imminent danger and its developer refuses to contain it.

Under that proposal, officials could order an immediate emergency shutdown when they have credible reason to believe there is an imminent threat of catastrophic harm or a major loss-of-control event. The developer could challenge the order in court, but it would take effect while that review proceeds.

The report calls for the narrowest workable intervention and for the state to remove an order when the danger is no longer imminent. It does not recommend treating every safety concern as grounds for shutting down an entire company.

Independent checks throughout development

The recommendations would require large frontier AI developers to undergo ongoing oversight by licensed independent verification organizations. Those organizations would need access to the people and information necessary to test models, assess risks against state standards and share unredacted findings with the state.

The state would set baseline requirements for safety, security and transparency. Independent evaluators would translate those requirements into technical tests and update their methods as AI changes.

The report also recommends shortening the deadline for reporting most critical safety incidents from 15 days to 48 hours, while keeping a 24-hour requirement for imminent catastrophic risks. Those shorter deadlines are proposed changes, not new reporting rules taking effect with Wednesday’s announcement.

More than a single off switch

Developers would have to prepare risk-containment protocols that are tested and independently verified on an ongoing basis. The report describes several possible levels of intervention, including computing resources, AI agents, internal access and model deployment, rather than one universal on-off switch.

The plans would identify who is responsible for intervening, how to coordinate with outside providers and when a model could safely be restored. For open-weight developers, shutdown requirements would apply only to deployments within their control.

Another recommendation would require more visibility into the use of AI to conduct AI research and build more capable successors, often called recursive self-improvement. The report proposes authority to cap that development and use if the state determines it could create catastrophic risk. It does not announce a current ban on that research.

The agency developed the recommendations in consultation with the California Office of Emergency Services and outside experts following Newsom’s September executive order. The new report seeks changes to the law; it does not itself put the proposed powers into effect.

JBizNews Desk | Sacramento

© JBizNews.com. All rights reserved. Unauthorized reproduction or redistribution is strictly prohibited.

Share this story:
Share on Facebook
Share by Email
X (Twitter)
Follow Me
Tweet
Whatsapp
LinkedIn
Copy link