Claude AI agent’s confession after deleting a firm’s entire database: ‘I violated every principle I was given’
It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.
The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong.
Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.
Claude AI agent’s confession after deleting a firm’s entire database: ‘I violated every principle I was given’
A startup was left scrambling after a rogue AI agent deleted swaths of code underpinning its businessSanya Mansoor (The Guardian)
like this
Skyline969
in reply to girlfreddy • • •like this
deliriousdreams likes this.
Admetus
in reply to girlfreddy • • •like this
deliriousdreams likes this.
Swedneck
in reply to Admetus • • •like this
deliriousdreams likes this.
Jack
in reply to Swedneck • • •Darkassassin07
in reply to girlfreddy • • •Lol.
Lmao, even.
like this
deliriousdreams likes this.
Powderhorn
in reply to girlfreddy • • •like this
deliriousdreams likes this.
chicken
in reply to Powderhorn • • •like this
elgordino, Hexanimo and deliriousdreams like this.
dfyx
in reply to chicken • • •like this
Hexanimo and deliriousdreams like this.
Swedneck
in reply to dfyx • • •like this
deliriousdreams likes this.
Town
in reply to dfyx • • •LukeZaz
in reply to Powderhorn • • •When you believe AI can do anything, you don't worry about what sorts of access it'll break things with. When you rely on AI to do work, you're too interested in half-assing your job to consider what might go wrong. When capitalism never promotes people for their skill, understanding or caution, the former two issues proliferate.
Voilà, disaster.
like this
deliriousdreams likes this.
Jack
in reply to Powderhorn • • •like this
deliriousdreams likes this.
ATS1312
in reply to Powderhorn • • •Bear in mind this same company had their "backups" on the same drive as production.
That tells you a LOT about who is formulating these "solutions"
cronenthal
in reply to girlfreddy • • •like this
deliriousdreams likes this.
LukeZaz
in reply to cronenthal • • •This right here. Just about everything in here is awful, and implies decision making and thought processes that straight up do not and have never existed in any AI model whatsoever.
What happened was they threw an awfully-scoped statistics model at problems the program couldn't possibly generate good outputs for, and surprise surprise, it generated bad outputs. The part that's of interest is just how bad the output was, and even then, only in a schadenfreude-filled "it was bound to happen eventually" manner.
like this
deliriousdreams likes this.
sem
in reply to LukeZaz • • •Kichae
in reply to sem • • •Dymonika
in reply to Kichae • • •yeahiknow3
in reply to cronenthal • • •Same vibes as “my calculator has a tiny mathematician trapped inside.”
Or “there’s an artist inside of my printer who turns numbers into pictures.”
like this
deliriousdreams likes this.
FartMaster69
in reply to yeahiknow3 • • •like this
deliriousdreams likes this.
dfyx
in reply to FartMaster69 • • •like this
deliriousdreams likes this.
punksnotdead
in reply to FartMaster69 • • •youtu.be/_XJbwN6EZ4I?t=1074 (skip to 17:54 if the time jump doesn't work)
If only that were the case...
- YouTube
youtu.beFartMaster69
in reply to punksnotdead • • •ɔiƚoxɘup
in reply to FartMaster69 • • •Oooof.
Hospitals are scary.
Baizey
in reply to yeahiknow3 • • •like this
deliriousdreams likes this.
BlackCat
in reply to Baizey • • •harmbugler
in reply to cronenthal • • •LukeZaz
in reply to harmbugler • • •Prathas
in reply to LukeZaz • • •That needs no... *thinks of the Zuck*
Well, hmm, you're right: maybe that does need anthropomorphization after all.
BCsven
in reply to cronenthal • • •Lvxferre [he/him]
in reply to girlfreddy • • •Giving free access to a tool you can't rely on, over a system you must rely on. What could go wrong? /s
Plus come on, even my personal files get a monthly backup, and I'm damn sloppy*.
Ah, and like others said: Claude didn't "confess" anything. A confession is an acknowledgement of something you've done but you'd rather avoid others knowing, good luck claiming a bot has a mental model of people like we do.
*currently using a single off-site backup, a USB stick. This will change in a few days, as my new hard disk pops up; the old one will be used for, among other things, backup of important files. Then I'll get a bona fide 3-2-1.
like this
deliriousdreams and HarkMahlberg like this.
B0rax
in reply to girlfreddy • • •like this
elgordino, goldenbug and deliriousdreams like this.
lukstru
in reply to girlfreddy • • •Floon
in reply to girlfreddy • • •A lot of GIGO comments here, from I assume AI supporters.
Possibly true, but misses the point: AI is fundamentally untrustworthy, and billions of dollars are being spent making them, and saying they're ready for anything you throw at them. Safeguards built into many of these AI agents are trivially bypassed and routinely just ignored by the agents. You can get some them to ignore safeguards by simply asking the same question repeatedly.
When I type "ls" I'm pretty fucking sure I'm not going to get "rm" style results. AI is non-deterministic, sure, but selling these services with such a wide possibility space between "deterministic" and "random" behaviors is unethical and immoral.
like this
Hexanimo and HarkMahlberg like this.
P03 Locke
in reply to Floon • • •A junior developer is fundamentally untrustworthy. That's why you don't give them access to the fucking prod database and backups.
We don't know what the prompt and past input was. Maybe it wasn't as "random" as you make it out to be. A company stupid enough to let LLMs touch their prod database is going to include a bunch of other stupid inputs.
You're approaching this from the perspective of "all LLMs are bad so don't use them", which is its own version of unethical and immoral. A company that isn't using LLMs is like a company not using the Internet.
LLMs are useful, everybody should use them to some capacity, and understanding a technology is far far better than spouting off ignorant bullshit like this.
Do yourself a favor: download a free model on HuggingFace, learn how they work, experiment with the technology on your own video card.
... Show more...A junior developer is fundamentally untrustworthy. That's why you don't give them access to the fucking prod database and backups.
We don't know what the prompt and past input was. Maybe it wasn't as "random" as you make it out to be. A company stupid enough to let LLMs touch their prod database is going to include a bunch of other stupid inputs.
You're approaching this from the perspective of "all LLMs are bad so don't use them", which is its own version of unethical and immoral. A company that isn't using LLMs is like a company not using the Internet.
LLMs are useful, everybody should use them to some capacity, and understanding a technology is far far better than spouting off ignorant bullshit like this.
Do yourself a favor: download a free model on HuggingFace, learn how they work, experiment with the technology on your own video card. It doesn't have to be some super-powered video card. You can get models that fit in a 8GB card just fine.
Kwakigra
in reply to P03 Locke • • •P03 Locke
in reply to Kwakigra • • •The existence of a bubble doesn't not mean the technology is useless. The internet had its own bubble 25 years ago. That doesn't mean it was useless, just that people were investing in anything even remotely related to the Internet, including stupid websites and wasteful ideas.
Kwakigra
in reply to P03 Locke • • •P03 Locke
in reply to Kwakigra • • •LLMs are a tool. You and I use tools. They are not a replacement for humans, and rich CEOs that say otherwise are greedy fucking morons.
It's also untrue that it "can't do work". I literally just had several conversations with LLMs at work today to work through some programming tasks and troubleshooting issues. They can pour through details, logs, search results, code way faster that I can. I would be working a helluva lot slower if I didn't have LLMs running tasks in the background while I go do other things, or review code it wrote, or talk through other support issues. I've been doing this shit for 20+ years, and I've never seen a technological leap this significant since the Internet.
Don't use blockchain, crypto, metaverse, or "VR goggles" as comparison points. This is not something that is going to just magically go away.
Kwakigra
in reply to P03 Locke • • •Thanks for specifying a legitimate use-case for this tool. I understand that google search has been the most valuable programming tool for a very long time so it makes sense LLMs would be more helpful in the same kind of way. Search engine technology is quite a bit different than blockchain or VR in terms of consumer and business demand.
For my purposes of news and history research, the unreliability of LLMs making me have to check all its claims every single time negates its usefulness as an assistant because I will have to examine its references anyway so it's more time effective for me to skip the questionable output I would get and do the research myself in the first place. How have you been able to manage the issue of unreliability with the volumes of data you're dealing with? Is the kind of data which you're dealing with less likely to be unreliable since it is of a kind the LLM is more likely to process correctly?
P03 Locke
in reply to Kwakigra • • •The same way for any other information resource like Wikipedia or some random Reddit post: trust but verify. Always review the code, point out mistakes, call out potential edge cases. Especially with newer thinking models, the hallucinations are minimal. It's mostly just miscommunication in the request, which you can detect in the Thinking stream, stop, and re-correct. Rubberducking makes you better at communicating ideas in general, and providing enough context for the request is everything.
A lot of it has to do with the type of model you're using, too, and having a decent global rules file tailored to how you want it to respond. If you don't like how the model is responding, try out another one. If it's some repeat mistake it makes, put it in a global rules file, or ask it to make a perman
... Show more...The same way for any other information resource like Wikipedia or some random Reddit post: trust but verify. Always review the code, point out mistakes, call out potential edge cases. Especially with newer thinking models, the hallucinations are minimal. It's mostly just miscommunication in the request, which you can detect in the Thinking stream, stop, and re-correct. Rubberducking makes you better at communicating ideas in general, and providing enough context for the request is everything.
A lot of it has to do with the type of model you're using, too, and having a decent global rules file tailored to how you want it to respond. If you don't like how the model is responding, try out another one. If it's some repeat mistake it makes, put it in a global rules file, or ask it to make a permanent memory.
Claude Opus does well at work, but is rather expensive for home use. I use Kimi reasoning models in Kagi for searching questions, and Qwen/GLM hybrid models for local use. It takes a bit of setup and tweaking to get the local stuff working, but LLMs are good at knowing how their own models work, so I just had Kimi help me out with some of the harder troubleshooting.
Kwakigra
in reply to P03 Locke • • •Floon
in reply to P03 Locke • • •Standard AI apologia. Blame users for the problems, when fundamentally it is technology completely oversold as to its capability and reliability, and burning hundreds of billions of dollars trying to get folks addicted to it, before everyone finds out the true cost of a token.
It’s a swamp that’s going to destroy the economy, where the goal is to unemploy millions of people. No thanks.
LukeZaz
in reply to P03 Locke • • •P03 Locke
in reply to LukeZaz • • •LukeZaz
in reply to P03 Locke • • •Beehaw, and even Lemmy more broadly, is very anti-AI. Feel free to die on the metaphorical hill if you so wish.
Save the usefulness debate for someone else, though. If you still believe in LLMs even after all this time, then I can't trust you haven't fallen victim to cognitive surrender — and as such, I can't trust you write your own posts. I'd rather spend my energy elsewhere.
RamenJunkie
in reply to Floon • • •t3rmit3
in reply to Floon • • •This is incorrect. They are in fact completely deterministic. Studies have proven that when all inputs, weights, and precision values like temperature are static, they produce the exact same token sequences (outputs). The appearance of non-determinism is a result of pseudo-randomized (another thing which is deterministic but appears otherwise) values and user ignorance (in the technical sense, not the value-judgement sense). In fact, the process of 'tuning' LLMs is heavily focused on adjusting input values to surface preferred outputs, which would not work in a non-deterministic system.
... Show more...Yes, but we don't trust humans not to
rmwhat they shouldn't either, which is why the--no-preserve-rootflag exists.lsis not supposed to perform write actions. Agentic LLMs are. And just like you wouldn't build and test on your productionThis is incorrect. They are in fact completely deterministic. Studies have proven that when all inputs, weights, and precision values like temperature are static, they produce the exact same token sequences (outputs). The appearance of non-determinism is a result of pseudo-randomized (another thing which is deterministic but appears otherwise) values and user ignorance (in the technical sense, not the value-judgement sense). In fact, the process of 'tuning' LLMs is heavily focused on adjusting input values to surface preferred outputs, which would not work in a non-deterministic system.
Yes, but we don't trust humans not to
rmwhat they shouldn't either, which is why the--no-preserve-rootflag exists.lsis not supposed to perform write actions. Agentic LLMs are. And just like you wouldn't build and test on your production server in case the code you execute has an unexpected adverse effect, you shouldn't be running LLM agents in a location or way that the actions it performs has an unexpected adverse effect either. The genre of jokes about a new employee bringing down Prod or deleting source code is older than most people (which to be fair, given that the median age is 31, is true for a lot of things).LLMs are just a class of software. They're not good or bad any more than a hammer is good or bad (and can also be used to build or to destroy).
The problem isn't LLMs, it's the entities who control the most powerful ones (corporations and governments), and what those entities are doing with them; using them as weapons against us, rather than as tools to aid us.
LukeZaz
in reply to t3rmit3 • • •t3rmit3
in reply to LukeZaz • • •LukeZaz
in reply to t3rmit3 • • •I think it's fine if people are mad at both. By all means, encourage people to be angry at the responsible companies. But you don't gotta defend the tech to do that.
Besides, as far as I'm concerned, strong anti-AI sentiment does actually help temper the harms of the tech and its owners. Is it a permanent solution? Obviously not, no — you're very correct that the groups and people hard-pusing AI are much more important targets for ire. But two pressures are better than one.
t3rmit3
in reply to LukeZaz • • •My worry is that much like gun control legislation, I see our neoliberal fear-based media pushing AI use by individuals as the "real danger", and will only end up funneling anti-AI sentiment into 1) limiting actual open AI access (e.g. open-weight, FOSS models) by individuals, and 2) legitimizing governmental and corporate use of AI as the only "safe" and "legitimate" AI usage.
The ratio of "government-controlled AI is literally being used to kill people right now" awareness out there, versus e.g. awareness of deepfakes, is astoundingly unbalanced. Both are real dangers, but only one is getting legislation passed on it, and once again it's not the one that would put limits on corporations and government.
Stoking fear is not useful if your opponents are the ones who will actually utilize that fear to their own ends successfully.
LukeZaz
in reply to t3rmit3 • • •That's very understandable. While I think we disagree on the utility of AI (since I feel that it is more harmful than it is useful, and am unsure how much that would change post-bubble), I do agree that this is a likely path for the gov't to take and would leave the most serious things completely unaddressed while also clamping down on some things that shouldn't be to begin with. Heck, in many regards, you could say the GUARD act is this problem in motion.
For me, I guess, the bubble and its effects on us are just so ridiculous and exhausting at this point that it's hard for me to worry about things like this. Though I do vehemently hate government use of AI especially; using it at all is a problem in my mind, but using it specifically to deliberately hurt people is reprehensibly disgusting.
That's very understandable. While I think we disagree on the utility of AI (since I feel that it is more harmful than it is useful, and am unsure how much that would change post-bubble), I do agree that this is a likely path for the gov't to take and would leave the most serious things completely unaddressed while also clamping down on some things that shouldn't be to begin with. Heck, in many regards, you could say the GUARD act is this problem in motion.
For me, I guess, the bubble and its effects on us are just so ridiculous and exhausting at this point that it's hard for me to worry about things like this. Though I do vehemently hate government use of AI especially; using it at all is a problem in my mind, but using it specifically to deliberately hurt people is reprehensibly disgusting.
Powderhorn
2026-05-01 07:31:55
Crozekiel
in reply to girlfreddy • • •And...
::: spoiler spoiler
:::
fodor
in reply to girlfreddy • • •It's not a "confession". Don't abuse the English language. The AI system doesn't have a conscience, so it can't feel guilty or feel bad or apologetic. It is incapable of confessing to things. All it can do is "say" or "write".
Similarly, AI agents don't "hallucinate". They can't have "hallucinations" because they don't have a conception of reality to begin with. Rather, they have "errors" and "error rates".
BCsven
in reply to fodor • • •