Comment by shwaj

From what I read here, the required chip size would scale linearly with the number of model weights. That alone puts a ceiling on the size of model.

Also the defect rate grows as the chip grows. It seems like there might be room for innovation in fault tolerance here, compared to a CPU where a randomly flipped bit can be catastrophic.