Comment by loyalcinnamon
4 days ago
As the hype is dying down it's becoming a little bit clearer that AI isn't like blockchain and might be actually useful (for non generative purposes at least)
I'm curious what counts as a midsize model; 4B, 8B, or something larger/smaller?
What models would you recommend? I have 12GB of vram so anything larger than 8B might be really slow, but i am not sure
My take:
Large: Requires >128GB VRAM
Medium: 32-128GB VRAM
Small: 16GB VRAM
Micro: Runs on a microcontroller or GPUs with just 4GB of VRAM
There's really nothing worthwhile for general use cases that runs in under 16GB (from my testing) except a grammar-checking model that I can't remember the name of at the moment.
gpt-oss:20b runs on 16GB of VRAM and it's actually quite good (for coding, at least)! Especially with Python.
Prediction: The day that your average gaming PC comes with 128GB of VRAM is the day developers will stop bothering with cloud-based AI services. gpt-oss:120b is nearly as good as gpt5 and we're still at the beginning of the AI revolution.
It can depend on your use case. Are you editing a large code base and will thus make lots of completion requests with large contexts?