Comment by robot-wrangler
5 hours ago
It's not about "every dollar spent" being a waste of time, it's about acknowledging the reality of opportunity cost. Of course, no one in any movement is likely to listen to their detractors, but in this case the pioneers seem to agree.
https://www.youtube.com/watch?v=DtePicx_kFY https://www.bbc.com/news/articles/cy7e7mj0jmro
I think there is broad agreement that new models and architectures are needed, but I don't see it as a waste to also scale the stack that we currently have. That's what Silicon Valley has been doing for the past 50 years - scaling things out while inventing the next set of things - and I don't see this as any different. Maybe current architectures will go the way of the floppy disk, but it wasn't a waste to scale up production of floppy disk drives while they were relevant. And ChatGPT was still released only 3 years ago!
And notably, Marcus has been banging this drum for years. Even this article points back to articles he wrote years ago suggesting deep learning was hitting the wall... With GPT 3....
It's sour grapes because the methods he prefers have not gotten the same attention (hah...) or funding.
He's continuing to push the ludicrous Apple "reasoning paper" that he described as a "knockout blow for LLMs" even though it was nothing of the sort.
With each of his articles, I usually lose more respect for him.