Comment by cedws
1 day ago
It's not a matter of knowledge, it's a matter of resources. It takes billions of dollars of hardware to train a SOTA LLM and it's increasing all the time. You cannot possibly hope to compete as an independent or small startup.
> It takes billions of dollars of hardware to train a SOTA LLM and it's increasing all the time.
True, but it's also true that the returns from throwing money to the problem are diminishing. Unless one of those big players invents a new, propriatery paradigm, the gap between a SOTA model and an open model that runs on consumer hardware will narrow in the next 5 years.
Eventually these super expensive SXM data center GPUs will cost pennies on the dollar, and we’ll be able to snatch up H200s for our homelabs. Give it a decade.
Also eventually these WEIGHTS will leak. You can’t have the world’s most valuable data that can just be copied to a hard drive stay in the bottle forever, even if it’s worth a billion dollars. Somehow, some way, that genie’s going to get out, be it by some spiteful employee with nothing to lose, some state actor, or just a fuck up of epic proportions.
at the point where those gpus cost pennies, they likely won't even be worth the electricity that goes into them, better models would run on laptops.
Presumably, the hardware to run this level of model will be democratized within the timeframe of the parent comment.
See https://amppublic.com and Stanford CS153, https://www.youtube.com/watch?v=mZqh7emiz9Q