Comment by grg0
2 days ago
As a SIMD noob, one thing that wasn't obvious to me is that SIMD can also speed you up if your mem throughput is underutilized by having the CPU load more data per instruction. It isn't just about compute speedups, which is typically what it's advertised for. Using perf on Linux has been very educational for me to get an intuition for modern CPU performance.
When designing CPU cores with SIMD execution units, it is very important to ensure that data can be loaded from and stored to the L1 cache memory at a throughput high enough to not starve the SIMD execution units.
Thus normally any CPU core that has a good SIMD throughput will also have a good throughput for transfers between the vector registers and the L1 cache memory.
The CPU cores of the last 3 decades that are considered well balanced can do the same number of FMA instructions per clock cycle (fused multiply-add) and vector register loads from the L1 cache memory. The throughput for stores is usually a half of that for loads, as that is typically enough.
For instance an AMD Zen 5 core in full configuration can do in every clock cycle two 512-bit FMA + two 512-bit loads + one 512-bit store (as a bonus, it can also do two 512-bit additions). A similar throughput is provided by the Intel P-cores used in their server CPUs.
If a CPU core would not have a memory load throughput proportional with the SIMD execution throughput, then much fewer programs would be able to reach a speed limited by the arithmetic operations, and most would be limited by the memory load throughput.
Even with the compromise currently chosen for designing CPU cores, the programs that use SIMD instructions are clearly divided into 2 classes, one where the performance is limited by the throughput of executing arithmetic operations and one where the performance is limited by the throughput of the load/store operations.
Makes a lot of sense, thanks for expanding.
The way simd speeds up "compute" is indeed mainly the reason that you operate on multiple pieces of data at once.
Yeah, and in the blog post he mentions that he had to transform the data to SoA. If he had done that alone, he might already have seen a speedup from better cache utilization.
Also, I see no mention of alignment in the post. I understand x86/AVX2 likes your load/stores to be aligned, even if it technically allows unaligned access.
Unaligned loads/stores aren't super bad; if still within a cacheline, there's zero penalty, and on crossing cachelines it's alike two ops (except page crossing, which is more bad).
So, for 32B loads/stores and 64B cachelines, it's 1.5x more L1 cache ops (as half of the ops will cross a cacheline); perhaps bad if you're L1-cache-throughput-bound, but less so if you're at L2+ as the extra work sits in L1.
If so they should just name it that
They should name it Multiple memory data, one single CPU instruction, or mmdosci for short. Easy to remember.
Well, the other way to say it is SIMD, "process more data in a vector at the same time", stops being advantageous when your "Von Neumann machine" cannot fetch enough data for the CPU in that time (the "bandwidth bottleneck" that has become a so pressing idea in the past few years...).