Comment by dragontamer
4 years ago
That's because there's really two conflicting goals to parallel programming.
1. Maximizing utilization of CPUs / GPUs / compute resources-- The "obvious" goal. You want as much code running in parallel as possible to accomplish some task faster.
2. Maximizing the utilization of SSDs / Hard Drives / Ethernet / I/O as much as possible -- Less obvious, but in I/O constrained problems, its not so much the CPU you're focused on, as much as it is the I/O you're trying to maximize.
Processes and threads are classically designed to solve #2, _not_ #1. Yes, we abuse processes and threads to make #1 go faster, but it really wasn't their original point.
When you perform a read() on a socket / Hard drive / whatever, it makes sense to "swap out" the process and find something else to do. This is optimizing #2, trying to run as many processes as possible to maximize the number of requests going to your I/O centers.
In contrast, if you're trying to perform a dense matrix-multiplication on AVX512 or GPU space or whatever, all this task-switching is completely useless and processes are detrimental to your goal, not beneficial. Its completely the wrong tool to use.
Bonus points: 4x GPUs working with a CPU (say, 64-core CPU) will run into both #1 and #2 problems simultaneously. Hurrah!
------------
Of course, today there's event driven code, coroutines, Golang threads, fibers, epoll... lots of tools to help you out on these tasks. But as the computer world grows more nuanced, it grows more complex. Its harder to figure out which tool to reach for in your toolbox.
Maybe there should be an IDE plugin that after you run your code once it makes a list of recommendations for parallelisation, including the possibilities "please rewrite this in language X and style Y" or "forget about it"
Intel VTune
> Processes and threads are classically designed to solve #2, _not_ #1.
Do you have a source for this?
Sure. The 1990s.
Single-core processors like the 486, Pentium, Pentium2 and Pentium4 had processes and threads, and many many programmers found them useful in the 1980s and 1990s.
Multicore computers weren't popular until the mid 00s, long after processes and threads were implemented in modern OSes.
------------
That is, for the entire time in the 1980s and 1990s, on single-core processors... processes and threads would _NOT_ speed up your programs (as per #1), because you only had a singular core. Task switching doesn't help at all on a 486 or a Pentium in terms of #1.
But task switching / threads helps in terms of Hard Drives (elevator algorithm), networking, and other I/O issues.
Even today: OSes like FreeRTOS offer you threads and/or processes on single-core systems like STM32G0 and other small microcontrollers.
----------
Windows 95 was when multiprocessing became popular in the Microsoft world. It would take another decade before typical home users had multicores. 99% of the time, a program sat there, waiting for the mouse to move or some other event to happen.