Comment by RajT88

My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents which exposed campaign finance violations and some truly nefarious stuff going on behind the scenes.

I did 90-95% of the work on this with a local GPU. I still ran Claude and Codex, but they were just writing code to orchestrate API calls against local models, local Whisper, etc. Doing the same with the public cloud would have cost thousands of dollars, instead it cost a few hundred dollars for a frontier lab subscription and tens of dollars for electricity.

I recently got into the DGX Spark and just a month ago, got an AMD Ryzen developer platform. Both are incredibly useful tools for some upcoming projects I'm working on (also related to government corruption), but in my limited experience of trying to run opencode on them, the "good models" are still completely unusable because the system prompt alone requires a minute of thinking.

I recently picked up R9700 to run Qwen 3.8-27B.

1) I have knowledge DB that I query, spend a lot of time to get "query N models, stream to me all N results, let me pick which one" - only to find out that qwen beat all other contenders (those were picked before I got R9700, so the rest is =<8B parameters)

2) Setup OpenCode to use it and gave it a few tasks:

   - first task (new feature to my MCP) took 40 minutes to complete with zero input from me, while it took 20 for sonnet-5 and sonnet-5 kept bugging me. Results are near identical.
 
   - second task (port a specific version of a package to my flake) it got stuck in a hilarious loop where model already been told what hash to use by nix itself, but it wanted to figure our how to get hash another way for some reason.

   - third task (another task, but much harder than first one with most of the discovery already done), kept doing discovery and running out of context, went through 3 compactions (256k context is what I can fit on R9700). Room got too hot, so I stopped it.

  3) virtual assistant like Hermes but my own: no notes, works great.

I'm pretty sure codding issues are just harness and lack of memory that Claude Code already had. Pretty nice setup, similar to mine but I built my own lightweight PaaS that is highly specific to what I run.

I have a similar setup to the OP (although I'm not sure if I'd call it a "software factory"), and I utilize a local model for some aspects of the setup. Specifically, I have a single RTX 3090 Ti with 24GB VRAM on my (main) home server, and I've been primarily running Qwen 3.6 35B A3B out of it via Ollama (I just switched to Qwen 3.8 27B, though, and I've also tried other Qwen models as well as Gemma models).

The Qwen models are decent, but they don't come close to the full Claude experience I've come to expect. As such, I only use the local models for specific tasks where it makes sense to do so. Really the setup is that my Claude-powered agents are able to incorporate my local model into work it builds out. The agents can perform inference against the Ollama API as they see fit, and I encourage them to do so for tasks where (a) the low-level capacity of the local models make sense and/or (b) where costs can become a concern.

It seems to work well when it comes into play (like having Claude drive a web browsing session but letting Qwen handle much of the actual browser interactions, image analysis, etc.). Still, Qwen just isn't smart enough (or fast enough on my machine) to handle anything agentic that isn't non-trivial.

Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS.

Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow.

The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking.

However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my server were cheap enough when amortised over several years.

[0]: https://github.com/Neroued/ninfer

We've put almost back in the title above.

(Submitted title was "A self hosted AI software factory")

Check out some NIXOS communities. People they are doing fun crazy stuff, as NixOS is immutable, thus creates separate sandboxes (building/taking down) per agent, etc.

I don't know most of the stuff there, but at least you can dive there.

"using Codex for inference" says the article