Comment by verdverm
14 hours ago
> Worth to mention: none of those findings overlapped.
This is why using multiple model families is important. Don't limit yourself by being a stan for this or that company.
14 hours ago
> Worth to mention: none of those findings overlapped.
This is why using multiple model families is important. Don't limit yourself by being a stan for this or that company.
Especially when $10-$20 on OpenRouter with e.g. DeepSeek (or any other inference provider, or their official API) will get you very, very far.
Gets me about a day or two lol
That may be the case, but simply running the same code review twice with the same model may give very different results as well. Impossible to know with the information we have.
yeah, there is definitely still a lot of vibes to all this still, I'm working on evals next to learn and leverage, to reduce the vibes component to the claims
today we can use some anecdata based on our observations, this also helps shape the dimensions we want to eval the agents-tasks on