Comment by SwellJoe
2 hours ago
I find that surprising.
I've been trying it on several projects and have found it's pretty sloppy. It leaves stuff broken, doesn't reliably write tests to check its own work unless explicitly prompted, misunderstands the assignment, etc.
It is smart and reasonably quick but not reliable.
I've come to the same conclusion over and over with all of the Chinese models that have been claimed to be catching up with OpenAI's and Anthropic's frontier models (Deepseek 4, GLM 5.2, Kimi K3).
At their best, I think they're closing in on Opus and GPT, but they're incredibly inconsistent and the variance in output quality is much higher than the best from any of the Anthropic or OpenAI models from the last few generations. The only way I can describe it is that it feels like a lack of intuition with the models which means I find my self needing to write longer prompts or have more back and forth to get them to do what I want from them.
To give an example, I have a saved prompt that I use as a sanity check on some data I'm storing. It reads about 50 rows from a DB and matches them to the UI and makes sure the data is displaying correctly. I've been using this with GPT 5.5 and now 5.6 for a few months and running it a few times a week with no issue. Sometimes I'll run it multiple times in a single chat if I notice bad data (run it, fix thing, run again, fix another thing).
I recently tried to switch to using Deepseek v4 (first flash and then pro) and while both did the task just fine, both would do things like change the response format from one message to another in the same chat or randomly decide to omit things it didn't think were relevant. At one point I ran the prompt, fixed some bad data, and then said "Okay, I fixed row 7, run {prompt} again" and so it decided to leave row 7 out of the response. A few times the first message would contain a table and then the next run in the same chat would contain the data in a bulleted list.
None of those are major issues and all could be solved with a bit more rigor in my prompting, but for me it makes them harder to work with. Those examples are a bit trivial, I think they're the easiest way for me to illustrate the gaps I see with them.
It’s because they’re doing some sort of combined score of intelligence, speed and cost. On pure intelligence it doesn’t even show up in the top 10.