Comment by mschuetz
12 hours ago
The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.
12 hours ago
The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.
Note that Vulkan is only somehow better than anything else out of Khronos, because at least for that API someone is paying LunarG for the Vulkan SDK, and NVidia has a vested interest in now pushing slang as the shading language.
The moment LunarG no longer gets sponsored it will be back to "here is a standard, all the best, the community will provide the tools".
On the SYSCL front, that is mostly driven by Intel, and their own OpenAI tooling, there are hardly any offerings for SYCL from anyone else. Thus it is really only pseudo-open.
This is at the core of the matter for me, and my knowledge is too weak to understand why this is the case. I don't enjoy the idea of relying on Nvidia's stack for GPU compute, but the alternatives I've tried (e.g. Vulkan compute) are higher friction to use. I am trying to reconcile why; Nvidia shouldn't have this moat.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
All commercial APIs have this moat, which is why other than Carmack famously advocating for OpenGL, the large majority of other devs never cared, they implement an API agnostic engine, use the best API for each platform, and move on with the rest of the game.
Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture.
See https://www.bit-tech.net/news/gaming/pc/carmack-directx-bett...
Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
3 replies →