Comment by adrian_b

I do not know about you, but I will probably also run this model, even with a mini-PC like an ASUS NUC, albeit very slowly.

Even if for your main LLM you want a speed high enough for interactive use and fast iterations, it is still useful to also run a much bigger LLM from time to time, even if that requires many hours for completing a task, due to the slow inference speed when the weights are read from SSDs.

There are tasks where the quality of the result is more important than the time needed to produce it. For example you might run all day a great number of iterations of a fast small LLM, and then run overnight a big LLM to check the end results of that work.

Therefore we need both small open-weights LLMs that can be run from the GPU RAM or at most from the CPU RAM, but also the biggest open-weights LLMs, even if the latter can be run only from SSDs by most people.