Comment by Someone1234
5 hours ago
You forgot to include whatever you're proposing instead.
"Trust me bro, Astra is better" isn't perhaps as useful as you seem to believe. I'm not even saying it is right or wrong, just that my opinion on this topic is still just one additional subjective data-point.
Only thing I wish with these benchmarks is that they would run repeat tests every couple of months. Then re-rank based on that too. We've seen a lot of performance fall-off after a couple of weeks with new releases.
Yeah unfortunately the benchmarks are usually provided by the company themselves, unquantized, thinking set to extra-extra-ultra-high, best of 10 runs, etc etc. It's hard to know how that's going to map to real world users.