Comment by nkrisc
3 hours ago
The AI part does seem relevant because it enabled incredibly low-effort “social” engineering.
For what it’s worth I don’t think you can call this social engineering since there was no human on the other end, even though it appears similar.
The question is, if there were actual human support agents, would they have built additional safeguards to prevent social engineering in this manner?
a human would have noticed something different about the requests it was getting, or the frequency of requests, and as soon as it noticed a shift, it would have carried that knowledge forward and intensified the scrutiny if something seemed off- eventually communicating it up the chain.
- instead of the ai context dying.
in the ai case, information only survives to the extent where the ai is empowered to store a note or notify a manager of an observation. Anything that does not result in sending a message/storage is wiped
Why did the account recovery system need AI. Surely just an email would do? What added value would AI add?
The person who writes the feature gets promoted for “aligning” with management's “Big Bets”.
There's no social engineering here, since all they have to do is copy and paste. This is a complete process design fail.