Comment by TimCTRL
7 hours ago
Yet none of them would have been here if Google hadn't published "Attention is all you need", the irony.
7 hours ago
Yet none of them would have been here if Google hadn't published "Attention is all you need", the irony.
The irony may only just be unfolding - check out this paper from 2015:
https://research.google/pubs/distilling-the-knowledge-in-a-n...
# LLM generated summary of the implied irony
Google may consider the standalone frontier-model arms race economically irrational, while still considering frontier-model capability strategically indispensable. Its longer game is probably not to avoid building the biggest models, but to build only enough of them to serve as capability factories—then turn that intelligence into a much larger population of cheap, purpose-built models.
(Human again) If Google knew what they were on to, why wouldn’t they make it their secret weapon from the start? I suspect it’s because they predicted there would be an arms race, and knew how to profit from it. They had a distillation paper published before “Attention is all you need”. In hindsight, is it ironic at all? Or is it obvious?
I think the answer is much simpler: They didn't recognize the potential of that specific architecture. I don't think anyone could have. They valued the perk to employees being able to publish research, and the prestige that followed, and googles ability to attract talent, more than they did the content of those papers.
Even Google wouldn't be here.
It was kind of like radiation science before WWII, it would be freely published because it wasn't potentially world changing yet. After it became a government interest, even people doing things unrelated to weapons would be much more apt to hold their work close.
The authors of that paper are all at AI companies not named Google.
They wouldn't be if Google hadn't published "Attention is all you need".
Lab employees do seem to be "knowledge sharing" as they move employers as far as I can tell, so I don't think that's true,
Oh that built on a bunch of ideas that weren’t far off. It was a great discovery but like many it’s standing in tall shoulders