Comment by smcleod
3 days ago
My 2023 Macbook Pro (M2 Max) is coming up to 3 years old and I can run models locally that are arguably "better" than what was considered SOTA about 1.5 years ago. This is of course not an exact comparison but it's close enough to give some perspective.
OpenAI released GPT-4o in May 2024, and Anthropic released Claude 3.5 Sonnet in June 2024.
I haven't tried the local models as much but I'd find it difficult to believe that they would outperform the 2024 models from OpenAI or Anthropic.
The only major algorithmic shift was done towards the RLVR and I believe it was already being applied during the 2023-2024.
I don't know about that. Even trying Devstral 2 locally feels less competent than the SOTA models from mid-2024.
It's impressive to see what I can run locally, but they're just not at the level of anything from the GPT-4 era in my experience.