Comment by tosh
4 hours ago
> My two favourite hypothetical questions regarding this used to be:
> If I'm running Codex and one of my API keys accidentally gets consumed in the context, what are the chances that someone else might ask for an API key in the future and get mine back? (I asked someone at OpenAI once and they called this the "regurgitation" problem and assured me that they take great pains to prevent that... but wouldn't describe how.)
> If I brainstorm with ChatGPT about potential new directions for my company, what's the chance that information might be exposed to a competitor in six months' time who asks "what might company X plan to do next"?
> My new preferred hypothetical for this is:
> If I use ChatGPT to help me partially solve a Millennium Prize problem, what are the chances that my work will influence training such that a later model helps someone else solve it first?
LLM can't be trained that easily. More like actual human are checking your logs and stealing valuable things from you.
Or searching anonymised logs for mentions of this problem and using that as part of the context or training.
This would work just as well and have plausible deniability.
Exactly! This is the real Occam's Razor explanation.
They wouldn't appear in weights but could be added to the context. My conversations regularly go "regarding your Java problem"... which was a separate item in the history from earlier. As long as I only see these (and nobody else sees mine), it can be helpful.
LLMs can learn from one sample.
Maybe just a rumor of a high value target having their API keys accidentally consumed in the context...