Comment by advisedwang

LLM's don't have any internal concept of "actual response" vs "surrounding words". Just like they don't have a internal concept of system prompt vs user input. Just like they don't even have an internal concept of what the LLM emitted vs what was given to it! It's all just one long sequence.

(Yes, it is possible to create tokens to represent category changes, but this is still in-band. the token is still just part of the sequence and the LLM isn't guaranteed to factor it in correctly)