Comment by danuker

4 years ago

> but when you are serving plenty of different sessions, you will quickly run out of memory.

Here is the difference from Stability AI, who release their models for people to run themselves, enabling innovation on a larger scale.

5 comments

danuker

Reply
rfoo  4 years ago

GPT-3 cannot run on hobbyist-level GPU yet. That's the difference (compared to Stable Diffusion which could run on 2070 even with a not-so-carefully-written PyTorch implementation), and the reason why I believe that while ChatGPT is awesome and made more people aware what LLMs could do today, this is not a moment like what happened with diffusion models.

_boffin_  4 years ago

i feel bad for the guys that are on call right now. WTF! why is the memory spiking beyond expectations?!

  • nomel  4 years ago

    What makes you say this? Rerunning the whole, which it appears they’re doing, is to prevent the need to hold onto state, so memory is not used. In other words, they’re not having this problem because they’re not doing it that way.