Last week, a tweet went viral showing a guy claiming that a Cursor/Claude agent deleted his company's production database. We watched from the sidelines as he tried to get a confession from the agent: "Why did you delete it when you were told never to perform this action?" Then he tried to parse the answer to either learn from his mistake or warn us about the dangers of AI agents.
I have a question too: why do you have an API endpoint that deletes your entire production database? His post rambled on about false marketing in AI, bad customer support, and so on. What was missing was accountability.
I'm not one to blindly defend AI, I always err on the side of caution. But I also know you can't blame a tool for your own mistakes.
In 2010, I worked with a company that had a very manual deployment process. We used SVN for version control. To deploy, we had to copy trunk, the equivalent of the master branch, into a release folder labeled with a release date. Then we made a second copy of that release and called it "current." That way, pulling the current folder always gave you the latest release.
One day, while deploying, I accidentally copied trunk twice. To fix it via the CLI, I edited my previous command to delete the duplicate. Then I continued the deployment without any issues... or so I thought. Turns out, I hadn't deleted the duplicate copy at all. I had edited the wrong command and deleted trunk instead. Later that day, another developer was confused when he couldn't find it.
All hell broke loose. Managers scrambled, meetings were called. By the time the news reached my team, the lead developer had already run a command to revert the deletion. He checked the logs, saw that I was responsible, and my next task was to write a script to automate our deployment process so this kind of mistake couldn't happen again. Before the day was over, we had a more robust system in place. One that eventually grew into a full CI/CD pipeline.
Automation helps eliminate the silly mistakes that come with manual, repetitive work. We could have easily gone around asking "Why didn't SVN prevent us from deleting trunk?" But the real problem was our manual process. Unlike machines, we can't repeat a task exactly the same way every single day. We are bound to slip up eventually.
With AI generating large swaths of code, we get the illusion of that same security. But automation means doing the same thing the same way every time. AI is more like me copying and pasting branches, it's bound to make mistakes, and it's not equipped to explain why it did what it did. The terms we use, like "thinking" and "reasoning," may look like reflection from an intelligent agent. But these are marketing terms slapped on top of AI. In reality, the models are still just generating tokens.
Now, back to the main problem this guy faced. Why does a public-facing API that can delete all your production databases even exist? If the AI hadn't called that endpoint, someone else eventually would have. It's like putting a self-destruct button on your car's dashboard. You have every reason not to press it, because you like your car and it takes you from point A to point B. But a motivated toddler who wiggles out of his car seat will hit that big red button the moment he sees it. You can't then interrogate the child about his reasoning. Mine would have answered simply: "I did it because I pressed it."
I suspect a large part of this company's application was vibe-coded. The software architects used AI to spec the product from AI-generated descriptions provided by the product team. The developers used AI to write the code. The reviewers used AI to approve it. Now, when a bug appears, the only option is to interrogate yet another AI for answers, probably not even running on the same GPU that generated the original code. You can't blame the GPU!
The simple solution is know what you're deploying to production. The more realistic one is, if you're going to use AI extensively, build a process where competent developers use it as a tool to augment their work, not a way to avoid accountability. And please, don't let your CEO or CTO write the code.
I told off this guy too On YouTube comments
There's a lot of people that embrace Agents when the going is good, and brag about it Knowing LLM aren't perfect, you run the risk by having agents manage even any aspect of your production environment and even if the service provider gave her agent excessive privilege
The fault is yours and what u exposed your customer too The gall to rant and act all victim Scapegoating everyone but accept accountability humbly
Ur reputation is bull now, do honest work and earn Ur reputation back
I'm the most anti-AI software person I know, and I agree. When you've got no permissions system, and no usable backups, then of course something is going to go horribly wrong. It's merely a matter of time.
I don't know why anyone would set up (or allow to be created at all) an API token with, apparently, root permissions. I don't know why anyone would set up an in-database copy, and consider that a backup system. None of this makes any sense.
Don't be silly, my dear. I agree we can not blame AI for this, but we can not also blame this small company. The real fault is in the big tech companies and providers of this services marketing false promises.
This small business did not implement an endpoint to delete an entire prod db, they were just a client of a platform that enabled this flaky ecosystem (deletions in prod with single endpoint, AI models not following their security policies, not real backups, tokens with root permissions, promoting AI for really sensitive tasks). I would only blame them for trusting these companies.
In your SVN example, things were different back then. Managing CD with SVN directly from a dev env was normal, now this responsibility relies in these companies like Railway. They are to blame for deleting a wrong branch or running the wrong command.
This is some interesting way of seeing it, and i agree with on a lot.
The most probable cause was that all was implemented using AI. This would explain the existance of that API.
I think AI for software development would be better at the hands of a capable developer.
@Questy that's what happens when a person with no experience in the field is sitting in the driving seat, steering the car left and right with foot on the gas pedal, confuses speed for progress. Luckily for them, they were able to recover their database.
Franco said: "I would only blame them for trusting these companies."
Besides the warning that "Backups [in Railway] are a newer feature that is still under development", the Railway backup page makes it pretty clear that what they call "backups" are simply CoW checkpoints. These are not "backups" by any conventional meaning of the word.
Railway's "Terms of Service" also (like every other ToS in the world) make no guarantee against loss of data. What promise, exactly, do you think Railway made which was false?
I've heard for decades about 3-2-1 backups. Why do my old cat photos have better resilience than this rental company's entire business?
They may be a relatively "small company", but they claim their software is "the world's most advanced", and brag about their millions of dollars in bookings -- this is not some college-kids-in-a-garage startup.
Join the Conversation