They share their methodology and results. I learned things about the relative strengths and weaknesses of Kimi and Fable I hadn’t seen anywhere else. Should being in the model hosting business disqualify them from sharing?
That seems to be actually a simple direct interest.
And, there is the "my product is good" confirmation to the outside incentive, there is the (only potentially) conflicting truth incentive, but there are also internal mission studies needs - so that you do not research into the benchmarks just to show people that "your product is good".
> call into question their results
That's always valid, so the question becomes "how much", and at that point the important side is difficult to evaluate - especially because in a frequentistic, Bayesian context the quantities were just potential anyway ("This will increase the chances - yes, of course just the chances - of E by some amount").
The don't only host open weight models. Also, why not promote this. If Fireworks thinks this big news might convert some new business doesn't make it not true.
They share their methodology and results. I learned things about the relative strengths and weaknesses of Kimi and Fable I hadn’t seen anywhere else. Should being in the model hosting business disqualify them from sharing?
Doesn't disqualify them, but it may call into question their results seeing as they have a potential conflict of interest.
> a potential conflict of interest
That seems to be actually a simple direct interest.
And, there is the "my product is good" confirmation to the outside incentive, there is the (only potentially) conflicting truth incentive, but there are also internal mission studies needs - so that you do not research into the benchmarks just to show people that "your product is good".
> call into question their results
That's always valid, so the question becomes "how much", and at that point the important side is difficult to evaluate - especially because in a frequentistic, Bayesian context the quantities were just potential anyway ("This will increase the chances - yes, of course just the chances - of E by some amount").
People are allowed to say, "hey we make money off of this thing, it's cheaper and almost as good as the thing we can't money off".
Other people are allowed to call them on that.
The don't only host open weight models. Also, why not promote this. If Fireworks thinks this big news might convert some new business doesn't make it not true.
and in fact, it would be detrimental to business, if they wrongly promote Kimi, as they would pretty soon lose trust with their users
I'd be doing the same thing if I were them as a marketing move
Anybody who's tried it knows that what they're saying is true though.