Comment by inciampati
10 hours ago
> We're not seeing them differentiate from each other, implying that the design space might not be that large.
There is the more likely reason they are not differentiating. They use almost exactly the same class of model. Everything is linear, parallelizable. It's incredible path dependence that's now invisible enough we think it's a natural law. Nature is not linear.
Exactly. They had the capital to gather massive amounts of data and compute and took a path that went well and pushed it to it's max. Are still pushing it. Are there better ways that don't need as much data and compute? Well, that's for the 17-year olds who don't have trillions to spend to figure out. I'm hoping they do so in a way that brings back affordable gaming gpus and memory sticks as a thing again.