Comment by throwawaymaths

there are several talks out there where Google soft-admits that at least the early gens of TPUs really sucked, e.g.:

https://www.youtube.com/watch?v=nR74lBO5M3s

(note the lede on the TPU is buried pretty deep here)