Comment by zozbot234
7 hours ago
Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K3 itself.
For software, not only that, but run batches with a cluster of agents working different parts of the same task. Software like Yegge's Gas Town has one agent act as "mayor," and others work on writing or testing various pieces, with all the agents messaging each other. In his book Yegge writes about using up to thirty agents at once.
I find it difficult to always have one or more long-horizon tasks 'queued up' and ready to run... I find myself usually bottlenecked on design, review, or something similar that requires me being in the driver's seat. It's possible I could queue up a bunch of tasks, letting the LLM run off in multiple directions, but then I'd be less able to steer and course correct.
Just my experience though, I'm still figuring things out. Perhaps some subsets of tasks would be more ideal for these long-horizon workloads - exploration, multiple competing implementations, etc...
Use your favourite harness to help you find long-horizon tasks to have queued up. It's changed the structure of how my projects work a bit, and do you have to do some homework fast of how slow/fast your various providers or local inference are, but it's worth it. Start off with hobby projects so you get a feel for how it works.
I left something gargantuan running over the weekend (decompiling 1980s-era system software) and look forward to checking it out later today when I have a few free minutes.
In principle, a slower inference ought to be easier to steer and course-correct. You'd always be able to look at partial results, especially with a local model that doesn't hide its thinking.
it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.
Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification from the chat frontend via RSS, Web Notifications API or similar that the full response is ready).
If your workload fits long batches throughout the night you could schedule them better, yeah. But I think very few have a usage pattern like that?
Given the hardware shortage in the world, I suspect renting ("sharing") via APIs will likely remain cheaper for the foreseeable future since each piece of hardware isn't sitting idle nearly as much.