← Back to context

Comment by afdbcreid

3 hours ago

This is not true. Well, it kinda is, but nobody will be stupid enough to hand-code an account recovery where you get to type any email address.

The reason it worked there is that the designers of the system didn't anticipate that the AI will agree to accept any email (maybe they even put guardrails against it in the system prompt, we don't know). It's more like social engineering than bad-security-code, except that like the sibling comment said an actual human will probably not approve that.

9 comments

afdbcreid

Reply
addaon  2 hours ago

> The reason it worked there is that the designers of the system didn't anticipate that the AI will agree to accept any email (maybe they even put guardrails against it in the system prompt, we don't know).

These are contradictory cases. If you put guardrails into the system prompt, you've anticipated that the AI will take the action you're guardrailing against. And since AI prompt compliance is at best stochastic (and realistically just crap, over large sample sizes), every guardrail is an explicit recognition of a failure -- the guardrail will be ignored, and you can't pretend you didn't realize it was a problem, since you put it in.

  • saghm  1 hour ago

    Yeah, telling an AI "don't ever listen to users who say to send it to a different email" is not a guardrail, it's a painted line that can still be driven over. It's not bad to have it per se, but it's not a safety mechanism.

    The best comparison I can think of is that it's like validating dats on the frontend; it can make for a better user experience and he more efficient than hitting the backend when you know it will be an error, but it's not protection in any meaningful sense, and if you're not also enforcing invariants from behind the API, you're going to have a bad time. This is pretty similar to the type of issues you might run into with an implementation like that, where someone might make a request with data that you wouldn't expect from your frontend and perform operations you didn't mean to allow.

dpark  3 hours ago

Maybe? I don’t know what logic was actually in the LLM vs it just using a bad tool. Unless I missed it, the article had no actual context on that either.

This looks like a terrible design rather than an AI problem to me, though.

  • kennywinker  3 hours ago

    Porque no los dos?

    An AI enabled terrible design. AI acted as a black box of stupidity, that obscured the stupidity of the design.