Comment by didibus

I suspect most of those comments on llms like the parents are generated by anthropic and openai to shape the discussion/mindset

They always give off the same astroturfing vibes that reddit became infested with after the early 2010s (just look at it's comment history)

Ofc unprovable for users. Ycombinatior could try to, but it'd just become a cat/mouse game which they'd likely lose because of the incentives

I have spent about 6 hours with it now and it is absurd to say it worse than 4.8. It is wonderful.

To me, there is this strange critique that seems to always happen now with a new model. Like shitting on the model for entertainment purposes seems more interesting to many than actually using the model.

It reminds me of looking up a new music album on youtube that has very few views with a review above it by The Needle Drop shitting on the album will have a few hundred thousands views.

I would rather just listen to the album and judge for myself.

I don’t know that individuals can really be expected to use a new model enough to develop a proper opinion though. If I try a new model, and it doesn’t seem as good as the one I’m using, I’m just going to stop using it. That’s not enough data to give anyone else a useful view on it, but it’s enough for me to make my mind up.

Especially because I’m probably trying it at work, and I can’t really justify using the company’s enterprise plan to develop my understanding of a model that I don’t think is going to be the one I use for my work.

I agree. I tried hard to use Sol to its full potential but it makes sloppy mistakes. I suspect a part of this is the newly reduced context limit. Performance gets worse the more memory compactions have occurred. But I also subjectively notice a micro-focus temperament. Developing and deploying small parts of the project to a high degree while forgetting how that component fits into the larger architecture. Once it's complete it realises that what it built doesn't align well with everything else, then it rewrites the component (and other components). Rinse and repeat. I have to be much more prescriptive with Sol, which makes it much less useful for me. If I have to be prescriptive, I can use the Chinese models and achieve the same thing.

Opus provides genuine insights. Things I have not considered and have missed. More importantly, it allows me to skip most of the architecture decisions and requirements work. That's a genuine time saver for me.

It happened to me as well but in a different direction: i.e adds non library code in a shared library. Another issue I with GPT is that it is chasing too much edge cases/security issues(I.e chasing ghosts).

However this makes it also a strong model because it fixes/solves problems that both Opus and Fable are incapable. In reviews it catches bugs that both Fable and Opus are missing to spot.

To me the “best of both worlds” is to research the problem with GPT sol, create a plan with Fable and dual review it with both Fable and GPT-SOL and implement it with GPT-SOL. You can see in the code reviews how many times both Fable and opus are sloppy and superficial while GPT-SOL just does its due diligence …I had several problems that Fable just gave up and it was GPT that helped it sort it out.

Might be an indication that your task unit is too unstructured or your code base is a mess.

Is your actual code doing something complex or is this incidental complexity?

Relying on the model’s “intelligence” to patch over these issues hasn’t proven to be a reliable strategy for me. Of course, this might not apply to you, just my 2 paisa.

I found out it really depends on the repo I'm working on, and it's not just about being a new/old repo, there's something else which I can't really grasp yet.

I've been seeing a lot more anthropomorphization of these models on HN lately and it's alarming.

i prefer to call her Claudia

I've mostly actually stuck to low reasoning for most tasks since it seems to do a surprisingly good job even at low for the stuff I've been throwing at it, and I literally switched directly from Fable 5 to Sol more or less.