Open weights are one thing, but honestly, the holy grail would be the original data (and some reasonably cost model by which to train something equivalent).
We've kind of reached the age of true industrial sized data. We've been acclimated to being able to eventually get to supercomputer levels of computation -- it will be a decade before we get there (if we ever do). It will be critical to retain the data that we have before it is locked off from the world into private vaults.
I think the other sibling response to this comment is reasonable: corporations violate the law and they pay some minimal penalty. people violate the law and effectively lose their life.
But if the original data is the whole internet (and then some), what would that even look like? Would it be useful to anyone? Are individuals going to re-run the training?
Open weights are one thing, but honestly, the holy grail would be the original data (and some reasonably cost model by which to train something equivalent).
We've kind of reached the age of true industrial sized data. We've been acclimated to being able to eventually get to supercomputer levels of computation -- it will be a decade before we get there (if we ever do). It will be critical to retain the data that we have before it is locked off from the world into private vaults.
I think the other sibling response to this comment is reasonable: corporations violate the law and they pay some minimal penalty. people violate the law and effectively lose their life.
But if the original data is the whole internet (and then some), what would that even look like? Would it be useful to anyone? Are individuals going to re-run the training?
It would look like a very large hard drive and a whatever the form of the cheapest way to train a huge model would be.
Useful? I guess without a ton of the RL training sets it may be challenging.