Hardware typically gets faster and cheaper over time. Unless we hit hard a wall because of physics then I don't see any reason that won't continue to be true.
The factories to make the better chips are themselves increasingly expensive; this is acceptable when their cost of construction can be amortised over more devices, but we're already at the point where the global poor get smartphones before safe water, so further factory cost increases can't really be assumed to be amortised better.
That said, current LLMs are not compute constrained, they're RAM and bandwidth constrained, so a (relatively) cheap factory that's dedicated just to filling a datacenter with hardware designed specifically for one particular AI architecture, that's something I think is plausible. As @tome accidentally reminded me about recently, the not-Musk Groq (https://groq.com/) is all about this.
Hardware typically gets faster and cheaper over time. Unless we hit hard a wall because of physics then I don't see any reason that won't continue to be true.
The factories to make the better chips are themselves increasingly expensive; this is acceptable when their cost of construction can be amortised over more devices, but we're already at the point where the global poor get smartphones before safe water, so further factory cost increases can't really be assumed to be amortised better.
That said, current LLMs are not compute constrained, they're RAM and bandwidth constrained, so a (relatively) cheap factory that's dedicated just to filling a datacenter with hardware designed specifically for one particular AI architecture, that's something I think is plausible. As @tome accidentally reminded me about recently, the not-Musk Groq (https://groq.com/) is all about this.
Well it's an analogy, that's a water-tight argument.