Comment by mindcrime

the reason why they're so inefficient is not algorithmic, but purely architectural.

I would agree with that, with the caveat that in my mind "the architecture" and "the algorithm" are sort of bound up with each other. That is, one implies the other -- to some extent.

And yes, fair point that building dedicated hardware might just be part of the solution to making something that runs much more efficiently.

The only other thing I would add, is that - relative to what I said in the post above - when I talk about "algorithmic advances" I'm looking at everything as potentially being on the table - including maybe something different from ANN's altogether.