Comment by jasonjmcghee
9 hours ago
The open source ones- I downloaded a number and tried them and compared to Jev.
Anything that required knowledge / familiarity mmBERT and ModernBERT post-trains performed much worse.
So it seems like they did some kind of useful expansive pre-training.
Things that were Qwen or Gemma Diffusion did better at those kinds of tasks but were generally pretty inconsistent in terms of whether they could succeed repeatedly (and be stable + reliable) on the many types of tasks that are in the cookbook part of the Jev docs.
If you ask Jev similar input + questions, it's pretty stable. And does a reasonable job on a lot of questions.
This one public benchmark (the only I've seen) seems to give the open versions way too much credit. It wasn't my experience at all.
It gave my a false wrong sense of what might be required to get it working for something at work to avoid needing a new subprocessor as - at least on Cloudflare / OpenRouter Jev is third-party not hosted.
Thanks for sharing your findings!
We are working on running our own benchmark of Jev and some of the other models. Our use case is classification that runs in a UI. Currently LLMs have good accuracy, but are too slow (and expensive).
Jev not being available through a cloud provider (Bedrock or similar) makes it more challenging for us to start testing and rolling it out.
I think I found Jev is now available through OpenRouter? Oh but maybe you mean somethign like Bedrock with zero-data-retention offered? Anyway, it was on OpenRouter the other day, lest anyone be confused by above, as I did some testing with it for my pareto frontier tool https://github.com/bglusman/model_skyline (which, apologies, is full of slop because its 100% AI maintained but, may or may not have some utility for guiding automated or manual decisions... it definitely showed that Jev was performing MUCH better than some of the open alternatives to it we tested anyway)
It's for data processing, privacy, and data retention. And not having to onboard a new vendor and data processor.
TypeSafe is still the provider. It's just proxied.
So it's a new subprocessor. Which can often be painful to onboard, especially if not compliant according to your needs.