Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill.
Kinda wish I had a Strix Halo here to play with as well.
I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.
Thanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more.
But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)
Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill.
Kinda wish I had a Strix Halo here to play with as well.
I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.
Thanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more.
But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)
2 replies →