Comment by roncesvalles
6 hours ago
I've found that capabilities are only "peaky" in GPT models (including 6 Astra). On Claude models (particularly Fable 5 and Opus 5.5), capabilities are fairly well-rounded. I'm astounded at how good Claude is at writing, storytelling, sociology, psychology etc.
I wonder if Karpathy has also let his opinion about frontier LLMs be informed a bit too much by OpenAI's models instead of exploring the other ones. It's not Coke and Pepsi at all.
At my own little sociology benchmark, if I'd rate Fable's performance at 7/10 (subjective scoring), Astra would be 2/10. They're that different.
Isn't Karpathy employed by Anthropic? He should be fairly familiar with the capabilities of recent Claude models.
I stand corrected. Then more broadly, there is a level in the "funnel" that comprises of people who have only tried ChatGPT and not Claude.
ChatGPT also has much stricter guardrails and will avoid going anywhere near large swaths of human knowledge.
I'd strongly implore anyone who uses ChatGPT for domains other than programming to try the same prompts in parallel on Opus 5.5 (Extra) for a few days. The difference is often staggering. ChatGPT is mostly just a coding and automation LLM.