Comment by pixl97
16 hours ago
>while having way less diversity of answers to that question than humans would.
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
>How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
> This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I think that this is most likely due to RLHF, as there's only a much smaller number of vendors here, and the likelihood is that the workers of the vendors preferentially prefer some kinds of actions.
I saw something like this in a previous life, where we had a small pool of human raters in cheaper places, which, when used as training data biased the models in a bunch of unexpected ways.
Didn't Elias come from WildChat?
I wasn't convinced by that explanation. Yes Cornell claimed that Elias the Lighthouse Keeper came from their WildChat dataset, which is a million conversations with GPT 3.5. But there are some problems here:
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
2 replies →