Comment by NikolaNovak
2 hours ago
As I said, my knowledge is very superficial - my background is relational databases and old school system administration, without much mathematical background since 3rd year linear algebra :-)
Fundamentally, LLMS are statistical and not deterministic. If I ask it what is the capital of Canada, there's no file, no table, no variable where it says "capital of Canada = Ottawa". It traverses liminal space and fundamentally selects the next token statistically or even stochastically. Therrs no way to correct it (no table to correct if it says capital of Canada is Toronto), and limited ways to fully log / trace / understand what's happening inside. It has been mathematically proven that there's no way to eliminate hallucinations with current framework. And prompt guardrails are best wishes.
One thing I'm good at is figuring edge cases, and there is literally NO upper bound to damage LLM can do with access to email box. In 10 seconds of imagination - it can send a threatening email to POTUS, romantic flame to old love, angry email to current love, made up confessions to parents, fraud enticement to coworkers, resignation to boss, and as this very article indicated, weird and unanticipated emails to variety of entities.
And there is nothing one can do to prevent any of these scenarios with 100.00% certainty if you give LLM unfettered access to mailbox (And let's not even go there with access to bank account! :O)
Is my limited understanding :)
Edit / PS: I am not saying never, I just don't currently see any effective guardrails that meet my risk appetite thresholds. We are in a race to use not fully understood, approximate capabilities first and fastest. In large percentage of cases it works great. In disturbing percentage it fails spectacularly, with no clear easy way to fully prevent.
I find it interesting that you are expecting a higher success rate (100.00%) for an LLM than you expect with many other things in your life with even costlier consequences.
You drive in vehicles that have a much lower than 100.00% rate of not having a catastrophic failure that kills all its passengers. Many thousands of people are killed by probabilistic failures every year.
Why must an LLM have 100.00% success before you would ever trust it with anything of value?
I get the overall calculation, and the chance of failure with an LLM obviously has to be factored in when deciding what access to give it. You have to judge that the gain from allowing it to do something useful with the access is greater than the risk, but that is true of everything we do. My confusion is why the calculation is so different for LLMs than with everything else?
Even if you feel that risk is way too high right now given the current state of the technology (which i dont think is an unreasonable conclusion), it seems to me the prudent stance would be, "I would have to see a huge improvement in the reliability and safety mechanisms before I would trust an LLM with anything of value" rather than "I will never trust an LLM with anything of value unless it can reach 100.00% success rate and a 0.00% chance of anything harmful happening"