Comment by lukeschlather
1 year ago
The alternative isn't to use a weaker model, the alternative is to solve the problem myself. These are all very academically interesting, but they don't usually save any time. On the other hand, the other day I had a math problem I asked o1 for help with, and it was barely worth it. I realized my problem at the exact moment it gave me the correct answer. I say that because these high-end reasoning models are getting better. "Barely useful" is a huge deal and it seems like we are hitting the inflection point where expensive models are starting to be consistently useful.
Yes, it seems we've only recently passed the point where these models are extremely impressive but still not good enough to really be useful, to now being actual time savers for doing quite a few everyday tasks.
The AI companies seem to be pushing AI-assisted software development as an early use case, but I've always thought this is one of the more difficult things for them to become good at, since many/most development tasks require both advanced reasoning (which they are weak at) and ability to learn from experience (which they just can't do). The everyday, non-development tasks, like "take this photo of my credit card bill and give me category subtotals" are where the models are now actually useful, but software development still seems to be an area where they are highly impressive but ultimately not capable enough to be useful outside of certain narrow use cases. That said, it'll be interesting to see how good these reasoning models can get, but I think that things like inability to learn (other than in-context) put a hard limit on what this type of pre-trained LLM tech will be useful for.