Comment by fooker

This will blow your mind, but it actually is pretty close to being unbounded. :)

Consider the 'MMA N matrices' primitive modern CPUs are starting to support. For the current generation of CPUs, N is a constant like 16 or 32, but there's nothing preventing it from being 1024 or larger if we have more memory bandwidth.

All this with a single instruction.