Comment by nradov
20 hours ago
All that compute is not needed. We have an existence proof from biology in the form of natural intelligence that much greater efficiency is possible. However, achieving dramatic improvements in compute efficiency will depend on unpredictable scientific breakthroughs. Personally I suspect that an entirely new hardware architecture will be needed, although I don't have any hard evidence to back that up.
>We have an existence proof from biology in the form of natural intelligence that much greater efficiency is possible.
It's only a proof that it's possible with 18+ years of training.
In certain ways my dog has more generalized intelligence than any LLM, and I trained her in only a few months with a modest investment in dog treats.
>from biology ... much greater efficiency is possible
Those are much more specialized models with pretty mediocre tokens per second.
Perhaps tokens is a dead end?
Perhaps! But perhaps whatever human brains use instead of tokens is not as amenable to scaling or copying.
[dead]