Comment by springtimesun
12 hours ago
I have never used sol, but I regularly use Claude to set up and benchmark local models per task. It is very thorough and has always returned good setups. The only thing you have to do is make sure you point at the model card. It’s always incredulous that models exist after its training cutoff.
K3 does an ok job of setting up, but its config searching isn’t nearly as thorough and its will confidently tell you it’s found the best setup when it’s only turned a few knobs. It’s also not a good evaluator of its own output. It rates its work too highly and seems kind of defensive when benchmarking. Still a good check because it does find stuff, but open a fresh session and don’t tell it where the results came from.
What K3 does do more than any other model I’ve found is investigate folder structures. If I want to benchmark it and other models I have to move the testing methodology and any reference to other results out of the project folder bc Kimi is like an ls bloodhound. It will find them.
>and its will confidently tell you it’s found the best setup when it’s only turned a few knobs.
on one hand , yeah : a model being more aggressive towards exploration of the decision space is usually a good thing.
on the other hand : I think that it's up to the operator to set rigid test criteria to make these things actually work well in a repeatable fashion.
so in other words, i'm glad claude is doing a better job for the way you're prompting the thing, but as a spectator from afar these kind of operator complaints usually spring up from the use of weak, ambiguous, under-considered prompts.
similar with the parents' complaint; there should have been a testing criteria for legible output that got immediately flagged or failed by the larger model.
it's a very hard sell for me to think that " a a a a a a a " is accepted as a valid language output by any near-SOTA-large model without some real coercion.
It’s not a complaint, only an observation. Operator aside, to observe that the models behave differently shouldn’t be controversial, should it?
It seems like you read my comment as dissing Kimi. Kimi is in regular rotation for me. Some tasks that Claude used to own go to Kimi first now. They are just different tools with their own strengths and weaknesses.