Comment by yearolinuxdsktp

I disagree with the naysayers. The README was scannable and sectioned so I could read what I wanted. But some things I wish were clearer, and I’m not sure they would be if you wrote it by hand.

Don’t waste your time rewriting it by hand just because keyboard warriors are up in arms on HN.

The material feedback here reflects the questions who might be from someone who’s been running local LLMs, so ask the LLM to update it with that in mind.

Another poster pointed out it’s not clear what the quantization is. There’s that one paragraph but it’s confusing. What I want to know right away is: are you running the unquantized model or is it quantized? If so, how much? Use terms like Q3 or Q4 or 4-bit or 8-bit. Explain why it’s not practical to quantize less. How much precision loss do you think there is at the quantization selected? What if I had 128GB RAM and wanted to have better precision and not higher token speed—would it be a good idea to choose 4-bit instead of 3-bit for some of these layers?