Comment by jtrn

I have been running it on my M5 Mac and was impressed with how well it worked with Pi coder. It can genuinely work as an assistant fully locally. It helps me configure Dockerfiles, fixed a couple of errors in a test Nuxt app, and so forth. Not very fast at 20 tps (8-bit quant for total memory usage around 30 GB), but enough to feel that I have a true local coding buddy.

Then came the cold water shower. The agent kept trying to figure out a Nuxt icon package issue and was working on it. On the positive side, it was making steady and slow progress without getting stuck in doom loops. But after 20 minutes, I decided to test with Luna. So I switched in Pi and asked it to review the problem and fix it. Same session. Thirty seconds later, it was fully fixed. API cost on open router was $0.02, probably most of it due to the inheritance of the previous session.

At that rate, the power consumption for local would be FAR higher than the API cost to solve the task.

I wish it wasn’t so, but the cost per intelligence is just off the charts now with Luna.

Now I am really liking that GLM 5.3 will probably run fine on 4x DGX Spark. If nothing else, the local models are truly usable for basic coding and assistance. I would have been blown away by the support I could have gotten with Qwen 3.8 when I was starting out coding. Hopefully, the local models will catch up AND the hardware becomes affordable in the future. Local models are keeping the largest LLM providers on their toes.

But right now, it does not make economic or capability sense to run locally. It does make privacy, security, and vendor lock prevention sense, though.