Comment by oblio

Up to now it’s basically that even when hardware gets more expensive, it also gets more powerful so the price per FLOP still improves. It might be debatable for mainstream GPUs, where generation gains are underwhelming, but the HPC versions are still making progress as long as you don’t need double precision.

I have no idea how the current spike is affecting this: the suppliers we deal with are adapting and we haven’t yet seen a new generation of consumer hardware since the RAM price surge (it’s not really a spike yet as we haven’t passed the peak).

When prices spike people generally try to make more of the thing so that they can profit from the high prices. It's very likely this is what will happen. The hardware for running inference will get very cheap and abundant, and the software will also get more efficient.