Comment by ryeights
5 hours ago
But an LLM provider could very easily add a "Jev mode" to any existing model, right? LLMs already produce a probability distribution over arbitrary classes. Just tell e.g. 5.6 Luna “here is the user's question, you must respond ONLY with the words 'foo', 'bar', or 'baz',” run a single forward pass of the model, and report the normalized probabilities of 'foo' 'bar' and 'baz' tokens before the first output.
With such an approach you could even retain full reasoning capability
My hunch is that you would need some post training. On top of that; I don’t think the llm itself can read inside the transformer state although I can see how that could be enabled. (I feel that would open up yet another class of exfiltration opportunities)