Comment by kouteiheika
7 days ago
> There was no such instruction. I've never chatted with any AI about LaTeX. it leaked from the tokens of someone else's chat.
Nope. That's not how it works. Attention doesn't work across multiple independent prompts queued in the same batch. It's not physically possible for the tokens of another chat to leak.
What most likely happened is that the model glitched out to the instructions in its (hidden) system prompt, which most likely does include instructions about using LaTeX for mathematical and scientific notation.
Maybe not due to attention, but it is certainly possible for chat content to get leaked into other conversations due to bugs in the stack, and in fact it has happened before.
https://openai.com/index/march-20-chatgpt-outage/
"We took ChatGPT offline earlier this week due to a bug in an open-source library which allowed some users to see titles from another active user’s chat history. It’s also possible that the first message of a newly-created conversation was visible in someone else’s chat history if both users were active around the same time."
You are probably right about this particular LaTeX issue though.