Comment by fwip
14 hours ago
At this point, it wouldn't be difficult to get a safety-trained LLM to prescreen your training set for the next model. (What that would cost, I can't estimate, but it seems simple in theory to reduce the amount of "harmful" training material).
Gemini Flash light is $.1/Million input tokens, Claude Haiku is $1/Million. Obviously input dominates here if it’s just a classifier. Training data easily can top 10 Trillion tokens - An earlier Kimi K2 was trained on 15T and even HF SmolLM 3B was trained on 11T.
So if I calculate right, it’s $100k-$1M per trillion tokens or $1-10M for a full dataset.
That’s way more than I expected, there is probably also some discount at that volume :)