Comment by zackmorris
8 years ago
I think so too, that SIMD is too low-level to be utilized effectively. Luckily joe_the_user enlightened me with this little gem on a previous post: https://en.wikipedia.org/wiki/Flynn%27s_taxonomy for explanations of terms.
Cool!
I have "evangelized" one person at least.
The thing about MOG is it's a demonstrated technique that Dietz has not yet developed into a fully releasable product. His latest comment says it's six months from release but lacks funding.
I think the problem might be that most "Mimd" programming is things like weather-simulations where access to a MIMD supercomputer is standard and price isn't that much of an issue.
That said, cheap Simd parallel computing jump-started today's deep learning advances and so cheap MIMD might do things no one anticipated.
Still, very niche. But thanks for the mention.
Edit: Note, MOG is specifically a GPU. Dietz did earlier work on other machines in the 90s but this is GPU specific (though a less flexible architecture than what Nvidia has evolved now).
what do you mean by effectively? Too hard for the programmer? Because what most of us find when we try to do SIMD stuff is anything higher level than intrinsics is too hard to use effectively where effective means good runtime performance.
I agree there should be something higher level we could use but it doesn't exist yet, to my knowledge.
Ya too hard for the programmer. What it comes down to is that there are a few main categories of multiprocessing from lowest to highest level:
1. SIMD - manually deal with packed elements (like in SSE, MMX etc)
2. DSP - Maybe someone knows a better term for this, but treating each slice of the data array as an independent serial stream (shaders, OpenCL, CUDA)
3. MIMD - freeform vector/matrix operations that get compiled down to the first two categories (MATLAB, GNU Octave, Scilab)
I'm not sure if TensorFlow fits best in 2 or 3 but my gut feeling is that it's closest to 2. The problem with the lower level abstractions is that it's more work to format the data for the problem space.
So with MATLAB, everything is a vector and operations applied to each vector happen in parallel across all elements. Then if you need to, you can drop down to less efficient code and operate on elements of the vector manually with C-like code.
Unfortunately that becomes more difficult in shaders, because you can't just magically access a neighboring element. And getting general computation to work with SIMD is often infeasible because you have to rewrite your code and potentially alter the layout of the data in memory to achieve better performance.
I also agree that currently nothing really exists to give us general-purpose vector math akin to MATLAB within a language like C.
The eigen c++ library comes somewhat close. But remember, Matlab is terrible at using multiple processors, so most people using Matlab won't see anywhere the performance most people c intrinsics get. This isn't because Matlab isn't capable, but I found that virtually no Matlab users try to write parallel code.