Comment by advisedwang
9 hours ago
LLM's don't have any internal concept of "actual response" vs "surrounding words". Just like they don't have a internal concept of system prompt vs user input. Just like they don't even have an internal concept of what the LLM emitted vs what was given to it! It's all just one long sequence.
(Yes, it is possible to create tokens to represent category changes, but this is still in-band. the token is still just part of the sequence and the LLM isn't guaranteed to factor it in correctly)
Thanks, so how is it possible for the Chatbot to box a Python response? Can't they use the same technique?
My understanding is: the training induces it to put Python code inside Markdown code fences, which the app interprets. It does that because the training data is full of examples of people doing that, because when people post Python code online it's very commonly in contexts where Markdown is understood that way.
... I suppose they could train the LLM to put a Markdown horizontal rule in the right place, but it sounds harder to get the system prompt to care about details like that consistently.