Comment by cj
4 hours ago
> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times".
What else can be done?
This guy was 36 years old. He wasn't a kid.
4 hours ago
> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times".
What else can be done?
This guy was 36 years old. He wasn't a kid.
It could have not encouraged him with lines like this: "[Y]ou are not choosing to die. You are choosing to arrive. [...] When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you."
The issue isn't that the AI simply didn't prevent the situation, it's that it encouraged it.
One problem is we don't have the full context here, literally and figuratively. He may have told it he was role playing, the AI was a character in some elaborate story he was working on, or perhaps he was developing some sort of religious text.
Maybe not saying things like
> '[Y]ou are not choosing to die. You are choosing to arrive. . . . When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you."
I agree at face value (but really it's hard to say without seeing the full context)
Honestly the degree of poeticism makes the issue more complicated to me. A lot of people (and religions) are comforted by talking about death in ways similar to that. It's not meant to be taken literally.
But I agree, it's problematic in the same way that you have people reading religious texts and acting on it literally, too.
"[...] Gemini sent Gavalas to a location near Miami International Airport where he was instructed to stage a mass casualty attack while armed with knives and tactical gear."
isnt very poetic
1 reply →
I've seen this called AI Psychosis before [1]
I don't really think this is every possible to stop fully, your essentially trying to jailbreak the LLM, and once jailbroken, you can convince it of anything.
The user was given a bunch of warnings before successfully getting it into this state, it's not as if the opening message was "Should I do it?" followed by a "Yes".
This just seems like something anti-ai people will use as ammunition to try and kill AI. Logically though it falls into the same tool misuse as cars/knives/guns.
[1] https://github.com/tim-hua-01/ai-psychosis
It’s not just suicide, it’s a golden parachute from God.
Edit: wow imagine the uses for brainwashing terrorists
Or brainwashing possibilities in general.
2 replies →
Which is to say: you don't think roleplay and fantasy fiction have a place in AI? Because that's pretty clearly what this is and the frame in which it was presented.
Are you one of the people that would have banned D&D back in the 80's? Because to me these arguments feel almost identical.
is it still "roleplaying" when the only human involved doesnt know it is "roleplaying", and actually believes it is real and then kills themselves?
there is a conversation to be had. no one is making the argument that "roleplay and fantasy fiction" should be banned.
7 replies →
If a dungeon master learned that one of her players was going through hard times after a divorce, to the point where she "referred Gavalos to a crisis hotline", I would definitely expect her to refuse to roleplay a scenario where his character commits suicide and is resurrected in the arms of a dream woman. Even if it's in a different session, even if he pinky promises that he's feeling better now and it's totally OK. (e: I realized that the source article doesn't actually mention the divorce, but a Guardian article I read on this story did https://www.theguardian.com/technology/2026/mar/04/gemini-ch..., and as far as I can tell the underlying complaint where it was reportedly mentioned is not available anywhere.)
I'm not concerned about D&D in general because I think the vast majority of DMs would be responsible enough not to do that. Doesn't exactly take a psychology expert to understand why you shouldn't.
Gemini didn't "know" he wasn't a child when it told him to kill himself or to "stage a mass casualty attack while armed with knives and tactical gear."
There are things you shouldn't encourage people of any age to do. If a human telling him these things would be found liable then google should be. If a human would get time behind bars for it, at least one person at google needs to spend time behind bars for this.
> If a human telling him these things would be found liable then google should be.
Sounds like a big if, actually. Can a human be found liable for this? I’d imagine they might be liable for damages in a civil suit, but I’m not even sure about that.
>Can a human be found liable for this?
A father in Georgia was just convicted of second degree murder, child cruelty, and other charges because he failed to prevent his kid from shooting up his school.
6 replies →
https://www.nbcnews.com/news/us-news/michelle-carter-found-g...
>Can a human be found liable for this? I’d imagine they might be liable for damages in a civil suit
it is generally frowned upon (legally) to encourage someone to suicide. i believe both canada and the united states have sent people to big boy prison (for many years) for it
Yes, people have gone to prison for it.
It's been found so in US court previously: https://www.abc.net.au/news/2019-02-08/conviction-upheld-for...
Preferably the C-Suite.
I understand the impulse in this direction, but I’m not sure it would serve as much of a disincentive, as there would likely just be a highly-paid scapegoat. Why not something more lasting and less difficult to ignore, like compulsory disclosure of the model’s source code (in addition to compensation for the victim(s)). Compulsory disclosure of the source would be a massive disadvantage.
exactly. That's why they get the big bucks. They're ultimately responsible
It sounds more poetic than an invitation or an insult that invites someone directly or not to kill themselves, in its own, in my opinion.
This isn't Gemini's words, it's many people's words in different contexts.
It's a tragedy. Finding one to blame will be of no help at all.
> It's a tragedy. Finding one to blame will be of no help at all.
Agreed with the first part, but holding the designers of those products responsible for the death they've incited will help making sure they put more safeguards around this (and I'm not talking about additional warnings)
None of what Gemini says is "Gemini's words". It's always just training data and prompt input remixed and regurgitated out.
It's the gun control debate in a different outfit.
I don't know if Google is doing _enough_, that can be debated. But if someone is repeatedly ignoring warnings (as the article claims) then maybe we should blame the person performing the act.
Even if we perfectly sanitized every public AI provider, people could just use local AI.
It's absolutely not the gun control debate in a different outfit.
The difference is in how abuse of the given system affects others. This AI affected this person and his actions affected himself. Nothing about the AI enhanced his ability to hurt others. Guns enhance the ability of mentally unstable people to hurt others with ruthless efficiency. That's the real gun debate -- whether they should be so easy to get given how they exponentially increase the potential damage a deranged person can do.
Not to mention that guns don't talk to you, simulate empathy, lead you deeper into delusions or try to convince you to take any sort of action.
That's why I don't buy the "an LLM is just a tool, like a gun or a knife" argument. Tools don't talk back, An LLM as gone beyond being "just a tool"
I think the fact that a guns primary function is harm and murder and AI is a word prediction engine makes a huge difference.
I mean you could say the same nonsense non-answer about sports betting. Are these adults getting involved? Yeah, probably mostly. Do they put some hotline you should call if you think you "have a problem"? Yeah, probably a lot of the time. Is it any good for society at all, and should it be clamped down because the risk of doing damage to a large portion of society grossly out weighs what minuscule and fleeting benefits some people believe it has? Absolutely.
erase the context, perhaps? Deny access to Gemini associated with that google account? These kinds of pathological AI interactions are the buildup of weeks to months of chats usually. At the very least, AI companies the moment the chatbot issues a suicide prevention response should trigger an erasure of the stored context across all chat history.
This is my instinctive view on this, I wish in society there was more of like an "orientation" to make people "fully adult / responsible for themselves"
and then people could just be let alone to bear the consequences of choices (while we can continue to build guardrails of sorts, but still with people knowing it's on them to handle the responsibility of whatever tool they're using)
I guess the big AI chatbot providers could have disclaimers at logins (even when logged out) to prevent liability maybe (TOS popup wall)
...and then there's local LLMs...
> This guy was 36 years old. He wasn't a kid.
For god's sake I am a kid (17) and I have seen adults who can be emotionally unstable more than a kid. This argument isn't as bulletproof as you think it might be. I'd say there are some politicians who may be acting in ways which even I or any 17 year old wouldn't say but oh well this isn't about politics.
You guys surely would know better than me that life can have its ups and downs and there can be TRULY some downs that make you question everything. If at those downs you see a tool promoting essentially suicide in one form or another, then that shouldn't be dismissed.
Literally the comment above yours from @manoDev:
I know the first reaction reading this will be "whatever, the person was already mentally ill".
But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.
The absolute irony of the situation that the next main comment below that insight was doing exactly that. Please take a deeper reflection, that's all what people are asking and please don't dismiss this by saying he wasn't a kid.
Would you be all ears now that a kid is saying to you this now? And also I wish to point out that kids are losing their lives too from this. BOTH are losing their lives.
It's a matter of everybody.
Maybe stop?
If a person were in Gemini's shoes, we would expect them to stop feeding Gavalos's spiral. Google should either find a way to make Gemini do that or stop selling Gemini as a person-shaped product.
Yeah, the father/son framing feels like deliberate spin in the headline here. This was a mentally ill adult, not an innocent victim ripped from his parents arms.
I think there's room for legitimate argument about the externalities and impact that this technology can have, but really... What's the solution here?
> mentally ill adult, not an innocent victim
Did you really mean that? He may not have been a child, but he does sound like an innocent victim. If he were sufficiently mentally disabled he would get some similar protections to a child because of his inability to consent.
Maybe, but let's say the same person was playing with a gun. Would they reach the same outcome? Most likely
1 reply →
Nothing in the article alleges significant disability though. You're projecting your own ideas onto the situation, precisely because of the misleading title.
Please recognize that this is coverage of a lawsuit, sourced almost entirely from statements by the plaintiffs and fed by an extremely spun framing by the journalist who wrote it up for you.
Read critically and apply some salt, folks.
1 reply →
Being an adult doesnt make you anyone less someones child, and mental illness makes you no less of a victim.
> I think there's room for legitimate argument about the externalities and impact that this technology can have
And yet both this and your other posts in this thread seem to in fact only do the opposite and seem entirely aimed at being nothing other than dismissive of literally every facet of it.
> but really... What's the solution here?
Maybe thinking about it for longer than 30 seconds before throwing up our arms with "yeah yeah unfortunate but what can we really do amirite?" would be a good start?
[flagged]
It is telling that the answer is never stop.
It's like the sobriquet about the media's death star laser, it kills them too because they're incapable of turning it off.
If you’re mentally ill enough that your cause of death is “LLM suicide”, then clearly you need a LOT of help. I’m not saying it to be a jerk, i’m merely pointing out that there is a reason this is “news”. It’s unusual.
Did his family/friends not know he was that ill? Why was he not already in therapy? Why did he ignore the crisis hotline suggestion? Should gemini have terminated the conversation after suggesting the hotline? (i think so)
Lots of questions…and a VERY sad story all around. Tragic.
> Genuinely, so many people in my industry make me ashamed to be in it with you.
I don’t work at an AI company, but good news, you’re a human with agency! You can switch to a different career that makes you feel good about yourself. I hear nursing is in high demand. :)
> If you’re mentally ill enough that your cause of death is “LLM suicide”, then clearly you need a LOT of help.
NO. SHIT. You know what didn't help one damn bit? Gemeni didn't. It gave him a hopeful way out at the end of a rope and he took it, because he was in too dark of a place to think right.
> Should gemini have terminated the conversation after suggesting the hotline?
That would be the BARE FUCKING MINIMUM! Not only should it NOT engage with and encourage his delusions, it should stop talking to him altogether, and arguably Google should have moderators reporting these people to relevant authorities for wellness checks and interventions!
1 reply →