Comment by butokai
4 hours ago
Do you think there will still be an incentive to release weights in that scenario? Everyone will have models only if there continue to be companies releasing weights.
4 hours ago
Do you think there will still be an incentive to release weights in that scenario? Everyone will have models only if there continue to be companies releasing weights.
Companies won't but I suspect this is a role that something else open source-y will fill that niche. Maybe orgs like wikimedia or internet archive, maybe some hackers just making things, maybe nation states that want to disrupt other players. Also model training will get better and better both on the algo and the hardware side. You can easily see a world where you might be able to train a good enough model on a home lab in a few days.
But you will need training data. Like a whole Internet search engine or massive data scraping. That‘s a thing that will not change with better algorithms, hardware or cheaper energy.
Data is the only moat but they'll be starting in the same place the current set of players statyed out just a few years ago. I suspect that the delta between what is publicly available (if not legally publicly available! see scihub) and what open ai and anthropic have is relatively small.