Comment by uejfiweun
1 month ago
This is pretty cool. How would you say that these open models compare to SOTA on coding tasks? I pay $200/mo for Claude Max but honestly this sounds way more fun.
1 month ago
This is pretty cool. How would you say that these open models compare to SOTA on coding tasks? I pay $200/mo for Claude Max but honestly this sounds way more fun.
Nowadays I use our local setup 95% of the time, but it is not that long since that flipped for me personally.
Context: I have a $20 Claude Code subscription, and have used it for a handfull of small-ish projects the last year, in parallel with local models on my AMD 9700XTX (24GB) at home. Mostly Ministral 14B and more recently Qwen3.6 27B Dense 4q.
Historically, the tooling (interferens engines and harness) has been the biggest challenge when using local models, a lot of the benefits from Claude Code was a rather unified and well oiled agent system. Local setups often bring with them sutle incompatibilities between models, inference engines and agent systems that are not obvious from initial testing, but cause trouble on projects larger than a couple of files.
The Spark setup at work is now at a point where I do not miss Claude, like at all. A big part of this is the harness and the tools available to the agent, most critically a good tool for searching online. I use my Kagi subscription to allow the models to fetch up-to-date information, and the Kagi MCP I use also has a summarizer which is very helpful in avoiding rapidly filling up the context window.
I mostly use Zed and it's native agent, which only recently got muuuch better, and on the terminal I use Pi with a minimal selection of extensions (currently pi-kagi-search, pi-smart-fetch, pi-btw and pi-diffloop). I also have Pi in Zed via the ACP, but it does not work so well with some of the extensions, especially the lack of a built-in permission system is a problem, when YOLO-mode is the only mode :)
Honestly, as long as you have a model that is decent at tool calling, your good. Having a solid and stable frame around your model makes a huge difference. The only caveat in all of this is that I spend most of my time on smaller projects and debugging on linux base systems, not huge and complex code bases, so your mileage might vary.
The next phase at work is to set up a chatGPT-like webinterface, and so far LibreChat is at the top of my shortlist. We had OpenWebUI for a while, but it is so bad at using MCP tools that it is practically non-functional for us. LibreChat is a bit more work to set up, but the interface and it's MCP story is much more solid. The goal is to plug in our internal helpdesk, docs and task manager system to LibreChat via MCPs to give us a quick way to query and gather information that is currently very time consuming to do on your own.