Comment by random__duck
5 hours ago
Opened the RTL, looked at the floating point math, learned that apparently you don't need correct floating point operations for LLMs, closed the page.
5 hours ago
Opened the RTL, looked at the floating point math, learned that apparently you don't need correct floating point operations for LLMs, closed the page.
Indeed there are some bugs/"non standard behavior" regarding very small or very big floating point values. All of those where proven harmless for LLM inference. Thanks for pointing out the bug. If you caught something outside of that, please point that out so I can fix it ;)
Well, you don't. That's why 1.58-bit (ternary) quants are often used. But if that's what they're going for, no need to dress it up in a floating-point facade.