Comment by HPsquared

This blog post [0] thinks that it is indeed possible to do arithmetic operations on them. My intuition is that they're vectors after all, and can be added and subtracted like any other vector. A word is just a location in that high-dimensional vector space.

[0] https://blog.dataiku.com/arithmetic-properties-of-word-embed...

EDIT: I guess there are different forms of word embeddings and apparently modern LLMs don't use static word embeddings like word2vec and it's more contextual. Tokens aren't 1:1 with words either of course. I guess it's more complex than "LLMs represent words as vectors". Still though it's a neat trick and is indeed that simple with something like word2vec.

Edit 2: some interesting slides including the bit about semantic analogy and a paraphrase from the original word2vec paper about the king queen thing: https://staff.fnwi.uva.nl/e.kanoulas/wp-content/uploads/Lect...

And the original word2vec paper by Mikolov mentions it https://arxiv.org/abs/1301.3781