Comment by lowbatt
1 day ago
If I wanted to run this locally at somewhat decent speeds, is an RK3588S board (like OrangePi 5) the cheapest option?
1 day ago
If I wanted to run this locally at somewhat decent speeds, is an RK3588S board (like OrangePi 5) the cheapest option?
Tried with S25+ (SD 8 elite). 0.82tok/s(4B L model). It's barely useful speed but it's pretty impressive either.
It depends on your idea of decent speeds and what you would use it for. I just tried it on a laptop with an AMD HX 370 running on battery in power save mode and it's not especially impressive, although it runs much better in balanced or performance mode. I gave it the prompt "write a fizzbuzz program in rust" and it took almost a minute and a half. I expect it to be pretty terrible on an SBC. Your best bet is to try it out on the oldest hardware you have and figure out if you can tolerate worse performance.
good idea, will test that out
RK3588 uses a 7 year old CPU design and OrangePi 5 looks expensive (well over $100).
A used sub-$100 x86 box is going to be much better
you're right. For my purposes, I was thinking of something I could use if I wanted to manufacture a new (smallish) product
I'm going to attempt to get it running on the BeagleY-AI https://www.beagleboard.org/boards/beagley-ai
Similar form factor to raspberry pi but with 4 TOPS of performance and enough RAM.