Comment by rafram
20 hours ago
This is a pretty obscure and in-the-weeds benchmark, but to me the models’ interpretation feels quite reasonable.
20 hours ago
This is a pretty obscure and in-the-weeds benchmark, but to me the models’ interpretation feels quite reasonable.
Apologies, I didn’t mean to imply it’s a benchmark, I just wanted to provide a reproducible example of where I see models make decisions that seem to be fine initially but might paint the software architecture into a challenging corner. I don’t expect models to read my mind, but I do see them produce a lot of verbose output, none of which is used to say “here’s a simple response to your ask, but have you also considered...”
It's obscure to use common functions from the standard library?
It’s just an odd way to judge whether an entire model “gets it” in my opinion. There are so many variables.
I think knowing which models use the standard library and which ones pull in dependencies as highly useful. As a dev I've always favored staying as close to the standard library as possible as it makes refactoring, understanding, and deletion much easier. Not everyone has the same preferences as you, it's nice to understand what matters for others too.