Comment by physicsguy
4 hours ago
A few gentle points:
(a) You mention that the NVidia docs push people to use libraries, etc. to really get high performance CUDA kernels, rather than writing them themselves. My argument would be that SIMD is exactly the same - they're something really that are perfect if you're writing a BLAS implementation but are too low level for most developers thinking about a problem to make use of.
(b) You show a problem where autovectorisation fails because of branching, and jump straight to intrinsics as the solution which you basically say are ugly. Looking at the intrinsic code, you're using a mask to deal with the branching. But there's a middle ground - almost always you would want to try restructuring the problem, e.g. splitting up loops and adding masks where there's conditions - i.e. lean into the SIMD paradigm. This would also be the same advice in CUDA.
(c) As you've found, GCC actually performs quite poorly for x86-64 optimisations compared to Intel. It's not always clear cut though, the Intel Compiler for e.g. sacrifices IEEE 764 float precision and go down to ~14 digits of precision in it's defaults, because it sets the flag `-fp-model=fast -fma`. This is true of both the legacy and new Intel compiler. If you switch to `-fp-model=strict` then you may find that the results are closer.
(d) AVX512 is quite hardware specific. Some processors execute these instructions much better than others. It's really a collection of extensions, and you get frequency downclocking that's better/worse on different processors as these instructions are executed.
Regarding (b), I would never rely on auto vectorization because I have no insight into it. The only way to check if my code is written such that auto-vectorization can do its thing is to compile my program with every combination of compiler, optimization setting and platform I intend to support, disassemble all the resulting binaries, and analyze the disassembly to try to figure out if it did autovectorization in the way I expect. That's a terrible developer experience; writing intrinsics by hand is much easier, and more robust. I'd need to re-check every piece of autovectorized code after every change and after every compiler upgrade.
I just treat autovectorization like I treat every other fancy optimization that's not just constant folding and inlining: nice when it happens to work out, it probably happens to make my code a couple percent faster on average, but I absolutely can't rely on it in any place where I depend on the performance.
> every combination of compiler, optimization setting and platform I intend to support, disassemble all the resulting binaries, and analyze the disassembly to try to figure out if it did autovectorization in the way I expect
I just used to fire up VTune and inspect the hot loops... typically if you care about this you're only really working on hardware targeting the latest instruction sets anyway in my experience. It's only if you're working on low level libraries I would bother doing intrinsics all over the place.
For most consumer software you want to be able to fall back to some lowest-common-denominator hardware anyway otherwise people using it run into issues - same reason that Debian, Conda, etc. only go up to really old instructions sets.
I work on games sometimes, where the goal is: "run as fast as possible on everyone's computer, whether it's a 15 year old netbook with an Intel Atom or a freshly built beast of a gaming desktop". As a result, the best approach is to discover supported instructions at runtime and dispatch to a function that's using those instructions (maybe populating a global vector function table at launch?). Next best is to assume some base level vector support (maybe the original AVX for x86, Neon for ARM) and unconditionally use those. Targeting only the latest instruction sets is a complete non-starter.
> (d) AVX512 is quite hardware specific. Some processors execute these instructions much better than others. It's really a collection of extensions, and you get frequency downclocking that's better/worse on different processors as these instructions are executed.
To re-iterate, this is our observation as well. The first AVX512 processors would execute such code quite fast for a short time, then overheat and throttle, leading to a worse wall-time performance than the corresponding AVX256 code.
I am not sure if there is a better way to find the fastest code path besides "measure on the target system", which of course comes with its own challenges.
The processors with severe throttling from AVX-512 were server CPUs, i.e. Skylake Server and its derivatives, like Cascade Lake and Cooper Lake.
Only few of those CPUs have been used in workstations, i.e. high-end desktop computers.
The vast majority of the CPUs with AVX-512 that can be encountered at the general population are either AMD Zen 4 and Zen 5 CPUs or some old Intel CPUs from the Tiger Lake, Ice Lake and Rocket Lake families. All these do not have AVX-512 throttling problems.
The owners of server computers are more likely to be knowledgeable about them and choose programs compiled with an appropriate target CPU model.
Therefore I believe that nowadays, when the percentage of computers with good AVX-512 support is increasing, and even Intel is expected to introduce by the end of the year Nova Lake with AVX-512 support, an application should be compiled such that whenever it detects AVX-512 support it should dispatch to the corresponding branch.
On the computers with AVX-512 support, using it can provide a significant increase in performance, while the computers where this could be harmful are more and more unlikely to be encountered outside datacenters that have failed to update their servers.
Skylake Server was introduced 9 years ago and Ice Lake Server, which corrected the behavior, was introduced 6 years ago. Therefore, wherever performance matters, the Skylake Server derivatives would have been replaced by now, as a single Epyc server can replace a cluster of servers with Skylake Server CPUs, at a much lower power consumption and with a higher performance.