Comment by theplumber
3 days ago
Opus5 is simply not as inteligent as Sol max. To me it looks worse than Opus 4.8 on some tasks. When I say worse I mean mainly superficial. I basically have to teach him how the whole app/framework works before he just jumps doing stupid stuff(I.e adding features already supported but in a different form)
Opus 5 hasn't been available for that long - long enough for benchmarks, but not really use and develop a subjective view on
I suspect most of those comments on llms like the parents are generated by anthropic and openai to shape the discussion/mindset
They always give off the same astroturfing vibes that reddit became infested with after the early 2010s (just look at it's comment history)
Ofc unprovable for users. Ycombinatior could try to, but it'd just become a cat/mouse game which they'd likely lose because of the incentives
> just look at it's comment history
I checked one. Old account, nuanced takes, and shitting on everyone equally. The perfect HN user!
1 reply →
That's not necessary. Sometimes we make choices and we feel the need to justify them in vivid flamewars regardless of how arbitrary. Vi vs emacs or amd vs intel, now anthropic vs openai.
We love to take sides, to belong to a small community of peers, no?
I have spent about 6 hours with it now and it is absurd to say it worse than 4.8. It is wonderful.
To me, there is this strange critique that seems to always happen now with a new model. Like shitting on the model for entertainment purposes seems more interesting to many than actually using the model.
It reminds me of looking up a new music album on youtube that has very few views with a review above it by The Needle Drop shitting on the album will have a few hundred thousands views.
I would rather just listen to the album and judge for myself.
I don’t know that individuals can really be expected to use a new model enough to develop a proper opinion though. If I try a new model, and it doesn’t seem as good as the one I’m using, I’m just going to stop using it. That’s not enough data to give anyone else a useful view on it, but it’s enough for me to make my mind up.
Especially because I’m probably trying it at work, and I can’t really justify using the company’s enterprise plan to develop my understanding of a model that I don’t think is going to be the one I use for my work.
Sol is a complete mess for me.
It only works on end to end tasks in fresh codebases.
Otherwise it cannot follow instructions, changes and deletes unrelated features or does sloppy work to mark a task completed while leaving a compromised codebase.
I could not get Sol to finish a feature in a complex code base without several loops of fixing and reverting
I agree. I tried hard to use Sol to its full potential but it makes sloppy mistakes. I suspect a part of this is the newly reduced context limit. Performance gets worse the more memory compactions have occurred. But I also subjectively notice a micro-focus temperament. Developing and deploying small parts of the project to a high degree while forgetting how that component fits into the larger architecture. Once it's complete it realises that what it built doesn't align well with everything else, then it rewrites the component (and other components). Rinse and repeat. I have to be much more prescriptive with Sol, which makes it much less useful for me. If I have to be prescriptive, I can use the Chinese models and achieve the same thing.
Opus provides genuine insights. Things I have not considered and have missed. More importantly, it allows me to skip most of the architecture decisions and requirements work. That's a genuine time saver for me.
It happened to me as well but in a different direction: i.e adds non library code in a shared library. Another issue I with GPT is that it is chasing too much edge cases/security issues(I.e chasing ghosts).
However this makes it also a strong model because it fixes/solves problems that both Opus and Fable are incapable. In reviews it catches bugs that both Fable and Opus are missing to spot.
To me the “best of both worlds” is to research the problem with GPT sol, create a plan with Fable and dual review it with both Fable and GPT-SOL and implement it with GPT-SOL. You can see in the code reviews how many times both Fable and opus are sloppy and superficial while GPT-SOL just does its due diligence …I had several problems that Fable just gave up and it was GPT that helped it sort it out.
But then Sol doesn’t really follow the plan or rather invents things it should add or cut on top of the plan. How does one deal with that?
1 reply →
Might be an indication that your task unit is too unstructured or your code base is a mess.
Is your actual code doing something complex or is this incidental complexity?
Relying on the model’s “intelligence” to patch over these issues hasn’t proven to be a reliable strategy for me. Of course, this might not apply to you, just my 2 paisa.
A well structured architecture, repo, and requirements document with concrete small deliverables can be competently delivered with cheap Chinese models. We rely on frontier models so that we don't have to spend several days/weeks on planning/architecture/documentation. That's their value proposition: superior intelligence. If they can't deliver that, they're useless at the current price.
I found out it really depends on the repo I'm working on, and it's not just about being a new/old repo, there's something else which I can't really grasp yet.
Yes it’s very codebase dependent, and changes over model versions. In my case codex was so bad, until it suddenly became better than Opus at 5.5, which surprised me. I am sure it can be the inverse for some codebases.
The way I keep an eye on model performance is alternating between models for code review. It shows how much the model understands the codebase without disrupting my workflow.
"him"? Have we reached that dystopia level?
I've been seeing a lot more anthropomorphization of these models on HN lately and it's alarming.
Not all HN visitors are native English speakers and in some languages "it" doesn't construct well with verbs, thus thought frameworks forms through usage of him/her. Nothing more to see I suppose.
It's a male name, and gendered pronouns can be hard for foreign speakers at times, irrespective of proficiency level. I wonder if you're overthinking this?
Every french person I've talked to IRL, for example, calls Claude "him." It's partly a language thing.
[dead]
i prefer to call her Claudia
I wonder if I'm secretly being routed to some low grade version of Sol, or any of the GPT models really. Their performance is outright insulting at times, even at maximum reasoning, yet if I were to only read HN, I'd never know.
I've mostly actually stuck to low reasoning for most tasks since it seems to do a surprisingly good job even at low for the stuff I've been throwing at it, and I literally switched directly from Fable 5 to Sol more or less.
Interestingly, if you filter by the coding index, Sol xhigh is the best one, and only Opus5 max is better than Sol max and Sol high.
My experience is the opposite. GOT 5.6 Sol is the dumbest most dangerous frontier model I’ve ever used. It actively introduces bugs and hacks and lies about what it did. Its code is almost always slop that can’t make it through even a brief review without half a dozen wtf moments. Opus is always cleaning up the terrible mess and GPT models are banned now from work because of how harmful and mind numbing stupid they are. R.e. Opus doing the wrong thing — my environment always gives the required context or leaves a paper trail so I don’t have that issue with Claude models.