Comment by gruez
18 hours ago
Cognition, the same company that a few years ago demoed a coding bot purporting to be able to autonomously complete upwork tasks, but upon closer inspection was going off the rails and not even completing what was asked?
https://www.youtube.com/watch?v=tNmgmwEtoWE
As others have mentioned this is post trained from Kimi k3, which is already quite capable, so it can't be that bad, but any claimed improvements in performance should be taken with a grain of salt.
A friend recently pushed me to try out their coding platform, Devin, after I decided to move away from Cursor. I had the same reaction: "What, the con artists from like 2024?" But after some cajoling, I gave it a shot and was pleasantly surprised. I guess they learned their lessons, grew up, and are doing good work now, maybe?
Their ads in SF are pretty funny. “Remember Devin? It’s good now”. Okay, very self-aware, Cognition.
Devin CLI is easily the worst-in-class coding harness I've ever been subjected to using, rife with bugs (up to and including dropping answers to the question tool), so I can't say I'm exactly inspired to try anything else the company produces.
Can't say I've tried the CLI. I've mostly focused on the cloud agents, which I was explicitly looking for. I compared against cursor's cloud agents, ampcode, and hoplite, and came out surprisingly enjoying devin.
I will say that the lack of parity between Devin cloud and Devin desktop is downright embarrassing. It's very clear that the latter is a thinly-reskinned Windsurf. A visually similar UI with vastly different capabilities. Definitely a black mark on the whole thing.
Had the same experience on it when it was going by Windsurf. Had to wire up a skill hooked to terminal runs or else it would hang or freeze and never finish literally every single time
I see issues with other harnesses too but not with the regularity I was getting from this. And the one moat they had with the better UI for per-project multi-agent orch disappeared and now is standardized
Not all that surprising. The original Devin really was just an early attempt at agentic coding before models were really even trained for it. Now that it's a well established pattern and we've figured out what works, I'm not surprised they've morphed into something reasonble.
When they launched Devin it was supposedly at the performance of an engineering intern. Friends who used it found the bad parts of an intern (tons of handholding, review required) but it didn’t learn from mistakes or add throughput.
They seem to love a good overpromise.
I think the evolution of the harness and ability to preserve loop context outside the context window has made running these kinds of agentic experiences easier.
sorry so many buzzwords to say, the capabilities to do this kind of work are more accessible and easier to manage, so now it works!
Good to see, and agree they were severely overhyping their product back then.
Devin lacks many features and doesn't even have an exec mode. Other than they subsidize the subscription, the added value here is low.
I used Windsurf for quite long and they're basically dead for now after they became Devin.
Everything feels dull and they're always several features behind while Cursor is just killing it every other week.
I'm back to VsCode plus Copilot Pro.
1 reply →
a reminder that money does enable you to make mistakes and buys you the ability to correct from them
> and buys you the ability to correct from them
Or at the very least, make more mistakes.
As others have mentioned, it's really matured a lot and at this point is one of the best cloud-hosted, team-managed coding agents, when factoring overall UX, testing and QA lifecycle via its sandboxes, and its ability to be controlled with an API. We use it quite heavily.
It's coming from a different starting place than Claude Code or Codex are as individually controlled single-developer tools. Devin has been more persistent in pursuing the direction of something that operates more autonomously at the team level, as a peer. And while it might be slightly behind in raw harness ability (maybe?) it's probably ahead on the team-focus.
Your startup is based around AI coworkers, so I expect you are biased towards overvaluing their usefulness.
Yes, perhaps fair. But my point was somewhat narrow. I wasn't saying that team-managed agents are good to go for all cases and that they're better than individual dev-managed ones. Just that they have gotten better and that of those Devin has some of the better UX.
Our experience might also not be typical because we have built infrastructure around making Devin and similar agents work better. And for the record no ties to Devin/Cognition. Just pay them too much as a customer.
I'm surprised they can post-train a closed model off of K3. Is that the norm for open-weight licenses?
Yes. If they are really open, you can use them as you wish. Some licenses are less open though.
I would be somewhat surprised if any license restriction actually holds up in court.
Well the models did get better but yeah their early product was godawful
Previous versions were based on Kimi too. I'd consider if it I could access the model outside Devin. No lock in for me, thank you very much.