Comment by schmorptron
9 hours ago
it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.
Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification from the chat frontend via RSS, Web Notifications API or similar that the full response is ready).