Comment by arw0n
15 hours ago
Strongly disagree with that take. Kimi K3 is a distilled model. I'm not saying that as a moral judgement, or to disparage the team behind it, but distilling and building on that is significantly easier and cheaper than building from the ground up.
And Gemini is kinda good enough at everything. Never the top, but it is decent at every task, and it is much faster than Kimi K3 and significantly cheaper. Kimi is very focussed on coding, Gemini isn't.
More importantly, it natively understands text, audio and video. If/when we are able to make the jump to robotics, this becomes essential. As you say, Google has a lot of deep background and deep pockets, they are able to make more of a long play. No idea if it will pay off, but it is way to early in the game to count them out.
You have some good points there.
Distillation does not make a model. There is so much more that goes into creating LLM Ai, traces from better models alone is insufficient. You need a good pre training run as foundation, a great RL reward function to make the other model's traces useful, and serious engineering chops to manage training runs at that scale. (non-exhaustive list)