Comment by sigmoid10
1 day ago
>I can't see how this problem becomes realistically solvable without hitting AGI
How would AGI solve this? The most common definition of AGI is "as good as average humans on all human tasks" - but in case of ITsec, that's a very low bar. We'd simply see prompt injections get more and more similar to social engineering as we approach AGI. Even if you replace "average" with "the best" it would still fall short, because human thought is not perfect. You'd really need some sort of closely aligned ASI that transcends human thought altogether. And I'm not sure if those properties aren't mutually exclusive.
That's a pretty recent definition, one developed out of marketing since it removes the need to further refine and allows it to be naïvely measured.
So I'll refine: sentient. I'll refine more: the ability to interpret the underlying intent of ill-defined goals, the ability to self generate goals, refine, reiterate, resolve and hold conflicting goals and context together, possess a theory of mind, possess triadic awareness. And I'm certain my definition is incomplete.
What I mean by AGI is the older definition: the general intelligence possessed by humans and other intelligent creatures. In context I mean much closer to a human than a cat.
>That's a pretty recent definition
It's actually one of the oldest definitions. I recommend you look up the works of H. A. Simon. This idea is quite ancient to people who are working AI research.
Anyhow, your more vague definition is still pretty much in line with my assumptions above in terms of the applicability to this issue. I.e. an AGI by your standard also will not bring a solution to this.