Comment by int_19h
14 hours ago
I'm genuinely curious as to what you're working on that you find 4B models good enough. I wouldn't even let a 27B model code, never mind supervising smaller models.
14 hours ago
I'm genuinely curious as to what you're working on that you find 4B models good enough. I wouldn't even let a 27B model code, never mind supervising smaller models.
At the moment I'm still doing shakedowns, so Typescript games compilation with a menu that has 4 games and retro artwork.
This seems to be a good example because things like the menu, high score boards etc are common, but the games are distinct. Then there's the artwork which requires decisions on look, and for coordinating.
The Qwen 4B model is multimodal so part of the AC is to view the output - I've a robust anti AI-look QA chain for that I've been using elsewhere, e.g. no floating parts, consistency, obvious missing fingers etc etc.
The longer term plan is to do some llama.cpp refactors specifically for some target hardware I have and implementing slightly different novel architectures I'd like to try (one I did already targeted CPU inference, which I did using 3 agents with specific roles; main planner, QA for planner, and benchmarking/environment handling)
The implementation was 85% of the speed of the original maxed out on my hardware but performance scaled with CPU core count whereas the original implementation plateaued. Unfortunately the break even mark seemed to be around 30 - non HT - threads.
I suppose I should look at that one again, since the increase in cores did not linearly drop off performance e.g. due to memory contention.