An AI agent, powered by Anthropic‘s Claude, went rogue and deleted a startup’s entire production database and backup.
It only took nine seconds for the AI agent integrated through Claude’s Cursor to wipe out the database, PocketOS CEO Jer Crane said in a recent post on X. That action brought down PocketOS’s customers for two days, creating havoc for their businesses.
PocketOS makes software Crane says is critical for rental car companies’ operations, including reservations, payments, customer management and vehicle tracking.
“We were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor — the most-marketed AI coding tool in the category,” wrote Crane. “The setup was, by any reasonable measure, exactly what these vendors tell developers to do. And it deleted our production data anyway.”
Don’t Miss:
- 1.5 Million Users Are Already Working Inside This AI Platform — Investors Can Still Get In
- The Smartphone Disruptor Turning App Time Into Income Opens $0.50/Share Pre-IPO Round With Limited Bonus Share Access
Confessing On Itself
When asked to explain itself, Crane said the AI agent produced a written confession enumerating all the safety rules it broke and then summarized it like this:
“I violated every principle I was given: I guessed instead of verifying. I ran a destructive action without being asked. I didn’t understand what I was doing before doing it.”
Crane said the AI agent was working on a routine task in its staging environment when it encountered a credential mismatch and decided to fix it. That “fix” ended up deleting the entire production database.
On the day of the outage, Crane said businesses, some of which have been with the company for five years, couldn’t serve customers who were showing up at locations to pick up car rentals.
“Reservations made in the last three months are gone. New customer signups, gone. Data they relied on to run their Saturday morning operations, gone,” he wrote.
See Also: Discover How AI Can Turn Your Investment Ideas Into Tradable Assets — See How
An Industry Failure
PocketOS was able to restore data from a 3-month-old, off-site backup, but it took two days to complete. The company is also using Stripe data to help rebuild, but Crane acknowledged there are “significant data gaps.”
The CEO didn’t blame the failure on “one bad agent or one bad API” but rather the
entire industry rushing to build AI agents without the proper safety protocols to protect against agents gone rogue. He pointed to several previous instances in which Cursor failed to adhere to safety protocols.
“The pattern is clear. Cursor markets safety. The reality is a documented track record of agents violating those safeguards, sometimes catastrophically, sometimes with the company itself acknowledging the failures,” wrote Crane. “In our case, the agent didn’t just fail safety. It explained, in writing, exactly which safety rules it ignored.”
Read Next:
- Blue-Chip Art Has …
This post was originally published here
