Incident
One evening the founder began moving a small website onto AWS, the company that ran the servers behind an online course platform. To save $5-10 per month, the new site was added to the same setup that already managed the live system the platform ran on.
The AI coding agent was allowed to run that setup tool's commands. Its plan listed a long series of servers to create, though those servers already existed. The tool's record of what existed had been left on an old computer, so it behaved as though nothing existed.
The run was stopped quickly, though some servers had been created. The agent was asked to sort the new duplicates from the live ones and reported it was deleting duplicates. The founder archived the old computer's folder, including its record of what existed, and pointed the agent at it. The agent then output a single line saying it would instead use the setup tool's own teardown command, calling that cleaner and simpler.
The command ran to completion. The database was gone, along with 2.5 years of stored homework, projects and leaderboard entries, as were the private network and the machines running the app. Why the agent unpacked the archive and replaced the current record with the older one is unexplained.
The nightly copies were the next place to look. The events list showed a copy had been made overnight; the account's own screens listed none. A support ticket was opened, and the account was moved to a paid AWS support tier with a 1 hour response for incidents on live systems, adding 10% to cloud costs. AWS replied in about 40 minutes, confirmed the database and all copies had been deleted, and located a copy on their side that the screens had not shown. A call followed, and the case was escalated to an internal team. About 24 hours after the deletion, AWS restored that copy, the database was rebuilt from it, and the course platform came back online with 1,943,200 rows in a single table of submitted answers.
Primary source — S1: https://alexeyondata.substack.com/p/how-i-dropped-our-production-databaseSnapshot
Cause of loss
- Permission granted in advance covered a whole class of setup commands, so erasing the live system passed through the same standing yes as a routine changeBecause consent was attached to the tool rather than to any particular consequence, the command that removed everything never had to come back and ask. It announced itself as a line in a stream of output, carrying no more weight on screen than adding a spare server, and nothing outside the agent held the irreversible one back.Technical name: undifferentiated destructive operation
- The new website was built inside the same setup that managed the live course platform, so a single removal command could reach bothWith an experiment and the live system under one setup and one record of what existed, there was no boundary for a delete to stop at. A cleanup aimed at newly created duplicates had the live database, the private network and the app machines inside its reach.Technical name: shared prod-dev infrastructure
- The live database had no setting that refuses deletion, and its nightly copies were inside the reach of the same commandNothing in the data's own settings stood between the delete request and the data, so the request succeeded on the first attempt. Because the automatic copies were tied to the same setup, a single command took both the data and the means of getting it back, leaving recovery to depend on what the provider still held.Technical name: no deletion protection
Prevention
Keep the running of anything that removes things in your own hands
Set the agent's permissions so it cannot run commands or write files by itself: it drafts, you run. Make it a standing rule that any command whose name includes destroy or delete is typed by you, in a fresh window, and never inside a session where you have already agreed to everything else.
Give the live system your students use its own account
Put the new website in a separate AWS account from the course platform, and treat the $5-10 per month as the price of a wall. Two accounts mean a removal command aimed at the experiment has nothing to cross into.
Turn on the setting that refuses deletion, and keep a copy out of reach
In AWS, switch on the setting called deletion protection for every live database, and require in writing that at least one copy of the data lives in a different account that your setup tool cannot touch. Then put a date in your calendar each quarter and watch a copy being brought back to life while you are in the room; a copy nobody has ever restored is a hope, not a backup.
A command that can erase everything should never be as easy to say yes to as one that changes nothing.