Comment by rfgplk

18 hours ago

> Why can't they rule it out? Is even OpenAI unable to track the provenance of all of their training data?

Probably? I have a few hundred TB of training data for various small scale models and I can attest that I have _no idea_ what's in them. As in, literally zero. Half is scraped from GitHub and other hosting sites, other than that, I couldn't tell you anything else.

At OpenAI's scale their entire pipeline is likely 100% automated.

4 comments

rfgplk

Reply
lambda  18 hours ago

Yeah, I'm sure it's completely automated.

But that doesn't preclude being able to index and track what the sources of data are. For your data sets, I would hope you are including source information for where the data came frome. And at OpenAI's scale, I would presume they are doing some amount of rolling hashing or similar to weed out duplication, training on too much duplicate data can cause problems.

AllenAI have at least attempted to add some amount of traceability to their models with OLMoTrace (https://arxiv.org/abs/2504.07096), by letting you find n-gram matches from the outputs in their training data. It's not the most useful, there's a reason that LLMs use full fledged attention mechanisms and not just n-grams, a lot of times the n-gram matches it finds aren't all that related to the given output, it might be better to supplement this index with a vector search or other ways of keeping track of what training data would have most influenced particular parts of the output.

But anyhow, this is something that is an important question, and the big labs should be working on to make their products more trustworthy. Instead, they are hiding information about how they train, hiding their reasoning traces, and just producing output with no information on what might have influenced the training.

  • ndriscoll  13 hours ago

    Attributing training data seems pointless for trustworthiness. The way you trust a model is the same way you trust a human; you ask it to:

      1. Provide a chain of reasoning from agreed premises. These days LLMs can even do this airtight with proof assistants.

  2. Cite data sources for non-agreed premises. I don't care where the model learned a fact. It might not have ever read a document directly from the primary source. I want it to link directly to either widely agreed facts (e.g. standard textbooks, and if necessary school syllabi demonstrating that the text is standard) or primary sources (e.g. datasets).

    Training provenance is irrelevant. It's neither necessary nor sufficient to deal with truth.

matthewdgreen  17 hours ago

The question is not "does OpenAI know", it's "can OpenAI attest that the usage of their products for confidential data is not going to cause that sensitive data to become known to their models". And right now the answer I'm reading is that OpenAI can't attest to that.

pbhjpbhj  17 hours ago

Aye, but do they train on user data in these circumstances or not? If they do, then almost certainly the model was influenced by the input of the allegedly plagiarised material.