Comment by mickael-kerjean

The infamous attention is all you need paper has this: "The Transformer ... reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.", the p100 is now quite a bit under 100$ and that's from a time where nvidia was valued 100x less than it is today. Lets wait and see