Comment by maherbeg
4 hours ago
Has anyone actually eval'd the other open source options against Jev on real world tasks rather than looking at benchmarks?
I see a lot of people parroting the quick open source alternatives as being better on the benchmarks, but it's such a new category that I'm not convinced we have solid benchmarks.
I'm hoping a company releases an internal eval benchmark for these options. I'm sure some of the open source ones are solid in some cases, but would love to see more reliable data.
I found this benchmark helpful: https://benchmarkheaven.com/jev-models
I also benchmarked them here [1] for content moderation. Jev is better than any of them.
[1]: https://tn1ck.com/blog/jevdit
This is awesome! Please submit updates when a new model comes out that people are saying are better than Jev!