Comment by mike_hearn
16 hours ago
The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
>How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
> This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I think that this is most likely due to RLHF, as there's only a much smaller number of vendors here, and the likelihood is that the workers of the vendors preferentially prefer some kinds of actions.
I saw something like this in a previous life, where we had a small pool of human raters in cheaper places, which, when used as training data biased the models in a bunch of unexpected ways.
Didn't Elias come from WildChat?
I wasn't convinced by that explanation. Yes Cornell claimed that Elias the Lighthouse Keeper came from their WildChat dataset, which is a million conversations with GPT 3.5. But there are some problems here:
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
Hum, I'm actually pretty sure that Text-davinci-003 used to talk about Elias all the time now that you mention it. I was doing stuff with NPC generation around Dec 22 and he was every third character.
1 reply →