Ask HN: Who is using FPGA for ML inference?
18 hours ago
With RAM price inflation, I wonder if FPGAs can be used to offload inference processing without keeping weights in RAM? The available RAM would be for activations, KV Cache, context but not static weights. Weights could be streamed from disk. This approach is not for tokens/second but throughput at a lower cost. Possibly better answers/kHh? I've started researching this, but wonder if others have considered/tried this?
I thought the same when I saw that the financial industry was hiring FPGA people for low-latency algorithms.
My understanding (as a non-FPGA expert) is that currently FPGA beats generic hardware (CPU,GPU) for "small size algorithm" (i.e that do not need GB of weights), while enabling a certain flexibility vs ASIC.
My guess is that you cannot bake all the weights into the circuit topology, so you are still bound by the memory transfer speed (to be double checked).
The value proposition I am pursuing is not faster speed, but increased throughput (and possibly reduced kWh).
I'm trying to understand the benefits of streaming expert weights through hardware that offloads the math and avoids storing all of the weights in RAM at once.
But that's not the only thing that can be streamed and offloaded. '
Finally, I'm trying to come up with approaches to reuse old hardware, old GPUs, old RAM instead of paying today's prices for GPU VRAM or unified memory. Even if I do not end up showing any particular FPGA benefit, I might be able to better run very large models on systems without GPUs or without unified RAM.
FPGAs are far more expensive than GPU+DRAM, even at today's inflated prices.
Tang nano FPGAs are around $10, the idea being to couple with a $25 ARM or RISC-V dev board and offload the math from a streaming store. Coupling this with an older 8GB VRAM board can theoretically run a model whose weights do not fit. The weights get repeatedly run through the MCU/FPGA and the VRAM is used for KV Cache and context. If you connected several via USB and streamed a MoE model expert per FPGA, you could achieve parallelism. Won't be fast, but, many requests could be processed in parallel, as each request shares the large static weights.
The reason for this is, model weights do not need to be randomly accessed. So why store them in expensive RAM.
Cerebras and Qrok seem to be using a very different approach than NVIDIA to get orders of magnitudes speed ups. I'm trying to explore other alternative approaches.
I predict that the FPGA adds no value in this scenario. Just process inference on the CPU.
1 reply →
You could build small ML model like talos-v2. But larger model/LLM requires much more engineering cost than you expected. Optimizing HDL has too many control knob/param to solve by RL.
Understood, and I like a challenge. I've also solved some tough engineering problems in the past (before AI). With AI it seems (and I emphasize seems) that I can tackle much harder problems. That's why I am proceeding to measurable experiments and not just taking AI chats as accurate.
I have a homelab, electronics lab, and software dev experience (electronics is a hobby, and an early career before I pivoted to O/S development). Before the recent RAM/GPU price jumps I had invested in a lot of used ECC RAM and many older GPUs (and some new GPUs) on many older servers that I refurbished and upgraded. It is hard to justify the prices of new GPUs/RAM going forward. It is cheaper to upgrade my CPUs (and I have) to have more cores.
Yes, my older hardware is slow by today's standards, but it is at least affordable. Yes, old hardware is power inefficient, but I justify that by using solar panels. I cannot do anything about the current supply/demand problems, but I can perhaps help with reuse and upcycling. Maybe my work will help students and junior programmers learn on old/used hardware.
Train circuits, not weights; then use an FPGA
Can you suggest what circuits? examples?
No one because FPGAs are 10-20x less dense than ASICs actually designed for purpose. The tooling is also complete shit. Signed someone that did (part of) their PhD on this topic.
But I can afford my already purchased drawer full of cheap FPGAs for experiments.
I guess I am too risk adverse to bet $10,000 on an ASIC run (having no experience doing that and no desire to go that route).
If I can demonstrate a proof-of-concept in a reproducible research way, others can then advance to the ASIC level.
This is just a hobbyist experiment looking for other hobbyists who can afford a cheap FPGA and have some free time and interest.