Comment by bertili
8 hours ago
Is there any (near future) technology that would permit burning this terrabyte into some kind of ROM chip?
8 hours ago
Is there any (near future) technology that would permit burning this terrabyte into some kind of ROM chip?
The path to ubiquitous AI (17k tokens/sec) https://chatjimmy.ai/, but it's running the rather outdated Llama 3.1 8B
Yes, from 6 days ago: https://news.ycombinator.com/item?id=48986351
That looks promising! As models become a commodity, this may turn out to be the real AI gold rush.
A question of course would be "is 15,000 tok/s Gemini better than 100 tok/s Opus 5"?
It's better for the billions of free users that Google serves.
Depends what you do. We have certain tasks we spend money on where Gemini 4.6 definitely is better than Opus 5.
Given how fast models are improving, burning the weights into actual ROM is prohibitively expensive if you need (or want) to replace the chip every couple of months.
The alternative is on-TPU flash for storing the weights.
Yes. We're still a ways off from this being ubiquitous, but I am convinced it's coming.