Comment by disgruntledphd2
4 hours ago
I'm not really convinced that there's much secret sauce here, all the methods and data are public, the only real difference is how much compute it takes.
4 hours ago
I'm not really convinced that there's much secret sauce here, all the methods and data are public, the only real difference is how much compute it takes.
All the methods and data are not public. We don't know what unpublished methods they're using. You can get most of the pre-training data publicly but they've probably spent a ton of money curating it and are now doing things like buying rare books. The RL training data is all (/mostly) proprietary though, and that's the real secret sauce part.
everybody do be cooking with water. Chinese Labs provided pretty good, primarily cost-reducing techniques, like the sparse attention patterns recently. I'd bet OpenAI and Anthropic use their variants of those too, so they can get greater margin on their tokens - not something they'd really want to / need to self-report.
This is obviously false. There is "secret sauce" because in fact not all the methods and data are public.
Where does the secret sauce show up in the outputs, then?
Like, (apart from tone), I find it hard to distinguish between the outputs of GPT/Claude/Kimi/GLM recently (I use cursor, and have been giving them the same prompt and comparing).
If anything, I found that the non-Claude models were better in many cases, which definitely doesn't map to their pricing.
> in fact not all the methods and data are public
Probably not, but unless you work at a lab, I'm not sure that anyone can say (and if you do work at a lab, you should not be replying on this thread).