← Back to context

Comment by _superposition_

12 hours ago

Great question. I suspect it's more than tokens.

Casually found this quote from Einstein, and personally it hits the nail on the head.

"The words or the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be "voluntarily" reproduced and combined. There is, of course, a certain connection between those elements and relevant logical concepts. It is also clear that the desire to arrive finally at logically connected concepts is the emotional basis of this rather vague play with the above-mentioned elements. But taken from a psychological viewpoint, this combinatory play seems to be the essential feature in productive thought—before there is any connection with logical construction in words or other kinds of signs which can be communicated to others."

  • That's the best evidence I have read so far for "Attention is all you need" ;)