Comment by leetharris
2 years ago
Has anyone found any success with Claude or have any reason to use it?
In my tests it is nowhere near GPT 3.5 or 4 in terms of reliability or usefulness and I've even found that it is useless compared to Mistral 7b.
I don't understand what they are doing with those billions in investment when 7b open source models are surpassing them in practical day to day use cases.
My experiences have been the same, unfortunately. It can do simple tasks, but for anything requiring indirect reasoning or completion of partial content from media (think finishing sonnets as a training content test) Claude just falls flat. Honestly, I'm not sure what makes Claude so "meh". Not to mention having to fill out a Google Doc for API usage? Weird.
This is just my two cents but I pay for both GPT and Claude as I find they complement each other.
I found Claude with the bigger context window quite good for doing "reviews" of multiple scientific papers, and answering questions about things like common findings or differences.
GPT couldn't do that natively at all until recently (and the few third party api-based solutions I tried wasn't good at it either), and just copy pasting text into GPT very quickly made it loose track.
Maybe the new bigger context for GPT means I can cancel Claude, but I haven't yet, going to give 2.1 a proper try first.
I also tried Elicit, and I believe they are on the right track, but did not produce anything useful when I tried.
I really think there is potential in using LLM's to for example do high level "reviews" of what is published on a specific topic but I have yet to find something that can do that. Claude with feeding it select papers manually is the closest. I hope someone at Google is building something around scholar.google.