Comment by teaearlgraycold
18 hours ago
I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.
18 hours ago
I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.
Depends on the actual audience, I guess. My stated “wisdom” comes in part from /r/LocalLLaMa, and my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side.
But there I literally did read “you don’t need anything better than 4 bpw” a bunch of times.
> my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side.
That sub is games, porn, and complaints about not having money. A waste of time.