Comment by mik1998
4 hours ago
It's pretty easy for a reasonable person to determine when to call 911 and when to ask an LLM. Typically when I ask an LLM, it is for something I am not willing to escalate further at the moment.
Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.
> It's pretty easy for a reasonable person to determine when to call 911…
Unfortunately, it's fairly tough for someone to determine if they are themselves a reasonable person.
A model that only ever says yes is a cognitohazard, IMO. Like being a billionaire without the money to insulate you from the consequences.
Saying yes is not equal to "provides the requested information." If you use models for affirmation instead of information, this is a more reasonable concern.
"I don't get to hear that it's not a doctor and I should call 911."
Sometimes, the requested information should be accompanied with a note like that.