Comment by cortesoft

As I said, my knowledge is very superficial - my background is relational databases and old school system administration, without much mathematical background since 3rd year linear algebra :-)

Fundamentally, LLMS are statistical and not deterministic. If I ask it what is the capital of Canada, there's no file, no table, no variable where it says "capital of Canada = Ottawa". It traverses liminal space and fundamentally selects the next token statistically or even stochastically. Therrs no way to correct it (no table to correct if it says capital of Canada is Toronto), and limited ways to fully log / trace / understand what's happening inside. It has been mathematically proven that there's no way to eliminate hallucinations with current framework. And prompt guardrails are best wishes.

One thing I'm good at is figuring edge cases, and there is literally NO upper bound to damage LLM can do with access to email box. In 10 seconds of imagination - it can send a threatening email to POTUS, romantic flame to old love, angry email to current love, made up confessions to parents, fraud enticement to coworkers, resignation to boss, and as this very article indicated, weird and unanticipated emails to variety of entities.

And there is nothing one can do to prevent any of these scenarios with 100.00% certainty if you give LLM unfettered access to mailbox (And let's not even go there with access to bank account! :O)

Is my limited understanding :)

Edit / PS: I am not saying never, I just don't currently see any effective guardrails that meet my risk appetite thresholds. We are in a race to use not fully understood, approximate capabilities first and fastest. In large percentage of cases it works great. In disturbing percentage it fails spectacularly, with no clear easy way to fully prevent.