Comment by user43928
13 hours ago
Do you guys actually work with these models?
I have to use GPT 5.4 Mini at work. It benchmarks higher than that Gemma 4 model.
In my experience it's next to useless. It cannot even move 20 existing lines of code from A to B without breaking them half of the time.
If you tell it to look something up in your dependencies, it's 50/50 on whether the answer is correct, incorrect, or it simply didn't perform the search at all.
I find it next to useless, and I'm mostly better off doing the work manually.
It's a night and day difference to even Sonnet, not to mention the SOTA.
“Moving lines of code” is a very peculiar eval tbh. I’ve never used Gemma for agentic tasks, but did have it write code, including multi-turn, and I was very positively surprised how well it performed.
It wasn't so much an eval, I really just wanted a small change moved out to another branch.
GPT 5.4 mini couldn't do it. Not even on the second attempt, where it went from obviously wrong to a subtly wrong copy.
In the end I had to manually copy and paste the 10-20 lines over.
If it can't even do that job, I seriously doubt it's going to be adequate for implementing a plan, like people often seem to suggest it could do, in order to save output tokens of a better model.
>It benchmarks higher than that Gemma 4 model.
Depends on what you look at. Gemma 4 31B without reasoning benchmarks significantly higher than GPT-5.4 without reasoning on artificial analysis. Even the new Gemma 4 12B beats it. And while GPT-5.4 with xhigh reasoning beats the reasoning version of Gemma 4 31B, the question is why you would throw such a complicated task that needs so much reasoning at such a small model to begin with. So if you do coding, you'll probably not have much success with either model. But for actual simple tasks that these models were made for, they are extremely capable. E.g. hook it up to the Atlassian MCP and have it do all the stuff that is supplemental to coding in big enterprises.
Counter: I use 5.4 mini all time for coding. No trouble letting it implement features. Entire new screens, APIs and various components.
It ain’t the best for sure, but if you have trouble letting it move 20 lines I don’t know what’s the cause but that’s not my experience at all. I do make pretty extensive use of guardrails and proper instructions in my AGENTS.md.
I also value super boring code bases with an as much as possible uniform shape. I guess that’s also helping out.
Like I said in my original comment, it’s fine for non-coding tasks, meaning I primarily use it to answer questions
Cursor 2.5 is essentially kimi and I find it eminently usable.
i use for tasks like object recognition in my family photos and cooking videos . seems to be fine
[dead]