Comment by Lerc
10 hours ago
>What if you mark the untrusted user input explicitly in the prompt,
I think the more robust approach would be to have whatever embedding vector the model attributes to untrusted input and to directly attach that vector after every layer of transformation. Set a mask of where to apply that vector programmatically for every external input.
That way it gets forced back into line if some sort of internal rationalisation tries to semanticly drift away .
From an interoperability perspective, this breaks the advantage of LLM inference that frontier AI labs have, in that you just have everyone run through the same algorithm but configure via text.
If you added probes at the model layer, you have to serve multiple different types of kernels at the same time, for multiple different companies and use cases (I guess you could provide a standardized set of probes for users), start tracking version control for each of the kernels, etc. very nasty compared to right now.
Could be a really interesting problem in the next 10 years or so, but this would require labs to be far more open about their models; and labs are still shooting for their AGI anyways, with the idea that nothing you suggest right now matters if AGI exists in a decade.
Exactly. I don’t have the spare time but have been thinking that even a bit mask about provenance and policy could be prepended to the vector, then training could reinforce adherence, including having output tokens that indicate the provenance of the inputs used for the token.
How does that guarantee anything? I could definitely see it being better, but that doesn't make violating it impossible does it? Just... statistically less likely.
Looked at that way, there are no security guarantees anywhere. Root CA’s can be compromised, cosmic rays can flip bits, zero days can appear in your supply chain.
Perhaps “ensure to a level ~six orders of magnitude better than current practices” would be a better way to say it.
RFC 3514 was just ahead of its time
Implementering the evil bit solves all our problems, I agree.