Comment by LarsKrimi

5 months ago

It seems very good at understanding human language clues even in a 4-bit (Q4_K_S) model, similar in feel to E4B but a great incremental improvement.

Interesting for my 8GB VRAM system, but the system RAM requirement seems to balloon quickly, and it starts misspelling words. Also token/s drops off quickly it seems