Comment by thot_experiment

2 days ago

I feel like I'm taking crazy pills with hy3, it's either benchmaxxed to hell and back or skill issue on my part but I'd rather use dense gemma. I don't think there's a single model that's wasted more of my time in recent memory.

5 comments

thot_experiment

Reply
SwellJoe  2 days ago

The Hy3 preview has been a mediocre performer in my benchmarks of security auditing with models, and yes, it is outperformed by Gemma 4 (31b soundly beats it, the MoE does slightly better, even at 4-bit quantization when using the QAT version). Qwen 3.6 27b also beats it.

I'll try it again now that it's out of preview and has been updated with more post-training. It presumably can't be worse, so maybe it's better enough to compete with a 31b model.

  • andai  2 days ago

    For context Hy is 295B, roughly on par with DeepSeek V4 Flash (284B).

    I haven't tested it yet so I cannot comment on the quality (nor the comparison with 10x smaller (!) models)

    • thot_experiment  2 days ago

      There are extreme diminishing returns in real world performance as models get bigger. 10x bigger might mean 5-10% better on benchmarks, a margin that can easily mean it's functionally equivalent in real world use or even a worse performer depending on the context it's being used in, and how good you are at providing meaningful context.

      Of course the bigger model embeds more knowledge, but when neither model has the knowledge necessary to perform the task, hy3 makes idiotic decisions all the time whereas gemma 31b has a decent hit rate.

      hy3 feels like someone who's read a lot of books and says the right words but has nothing of substance between their ears, gemma feels like a reasonably intelligent person who doesn't understand the domain, the latter is muuuch easier to work with than the former.

      1 reply →