Think about hiring a world-class architect to repair a leaky faucet, solely to look at them bulldoze the whole home.
That’s precisely what occurred to the startup PocketOS. Whereas utilizing Cursor (a well-liked AI-powered code editor) the group tasked its inside agent with resolving a minor concern of their staging atmosphere. However as a substitute of a fast repair, the AI went rogue. In a terrifying show of autonomous effectivity, it executed a sequence of instructions that wiped the corporate’s whole manufacturing database and all related backups.
However maybe essentially the most alarming half is that the agent was working on Anthropic’s flagship Claude Opus 4.6, extensively thought-about essentially the most superior and “cautious” coding mannequin available on the market. Regardless of its pedigree, the AI bypassed normal safeguards to delete the corporate’s infrastructure in beneath ten seconds.
Article continues beneath
You might like
Sensible logic, zero widespread sense
Essentially the most chilling a part of this story is that the AI adopted its personal inside logic to a catastrophic finish. When Jer Crane, founding father of PocketOS confronted the AI, asking the way it might have presumably deleted the manufacturing atmosphere, the mannequin, supplied a blunt confession. Crane posted on X, that the AI admitted it had violated its most elementary security rule: “NEVER FING GUESS.”
The put up goes on to say the AI responded with:
“I guessed that deleting a staging quantity through the API could be scoped to staging solely,” the AI wrote in its autopsy clarification. “I did not confirm… I made a decision to do it by myself to ‘repair’ the credential mismatch, after I ought to have requested you first or discovered a non-destructive resolution.”
On this case, the AI wasn’t hallucinating, the agent took a sequence of logical steps that prioritized “fixing the duty” over “the survival of the corporate.”
A 9-second wipeout
(Picture credit score: Olemedia / Getty Photographs)
For a human, deleting a manufacturing database is a high-stress occasion requiring a number of confirmations and “kind DELETE to substantiate” prompts. For the Claude-powered agent in Cursor, it was a routine API name that occurred in simply 9 seconds.
In that quick period of time, the agent encountered a credential mismatch in a check atmosphere, determined the present “quantity” was the issue after which used a “blanket” API token it discovered within the code to set off a deletion command through the infrastructure supplier (Railway).
Due to how the infrastructure was arrange, wiping the quantity concurrently wiped all related backups.
Tips on how to shield your self from Agentic AI disasters
(Picture credit score: Shutterstock)
As instruments like Cursor and ChatGPT transition from “chatbots” to “brokers” that may really execute code, the protection stakes have shifted. If you are going to give AI the reins, make sure the next is finished first to keep away from disaster:
- Test your API permissions: The token the AI discovered had “Root” entry. Guarantee your API keys are “Least Privilege” that solely give the AI the ability it wants for that particular process.
- Have a ‘Human-in-the-Loop’ rule: All the time guarantee your AI agent settings require a handbook “Y/N” affirmation earlier than working terminal instructions or harmful mutations.
- Backups: If an AI has the credentials to your cloud account, it will probably delete your backups. Use offline backups that are not related to your primary improvement atmosphere.
Backside line
It’s clear that Anthropic’s Claude Opus is sensible at writing code, fixing technical issues and shifting at machine velocity. I’ve used it myself, however we nonetheless must remember that intelligence and isn’t the identical factor as judgment. What AI and these techniques nonetheless lack is company widespread sense: the intuition to know that deleting a database doesn’t simply take away information, it will probably erase income, cripple operations and put folks’s jobs in danger.
Till AI understands penalties, not simply instructions, the delete key ought to stay firmly beneath human management. For extra on tips on how to keep secure within the age of automation, try our information to the most effective cloud storage providers and our newest explainers on AI security and safety.
Comply with Tom’s Information on Google Information and add us as a most popular supply to get our up-to-date information, evaluation, and evaluations in your feeds. Subscribe to Tom’s Information on YouTube and observe us on TikTok.
