Comment by YetAnotherNick

It's not just scale. Even for single GPU, it is hard to acheive 2x speed improvement as the GPU specs states. Even NVIDIA's own Tensor Engine acheives 28% extra FLOP/s[1].

[1]: https://arxiv.org/pdf/2310.18313