Comment by axg11
4 years ago
Thanks for your perspective. We’re still in disagreement but I wouldn’t bet on either side of the AGI debate with any significant conviction.
Embeddings are very good at a few things: combining concepts (addition), untangling commonalities (subtraction) and determining similarity between concepts (distance).
> While embeddings are a great tool for compressing information, they do not provide inherent mechanisms for manipulating the information stored
What are the manipulations you’re referring to? I would love to learn more. From my understanding, embeddings actually provide great generalisation. If you have a well conditioned embedding space then you can interpolate into previously unseen parts of that space and still get sensible results. That is generalisation to me. Many current ML methods do _not_ result in a fully meaningful embedding space but my hunch is that we will get there with future insights and advances.
> We’re still in disagreement but I wouldn’t bet on either side of the AGI debate with any significant conviction.
That is probably a superior position to hold. I am agnostic by nature, and interestingly this is one of the rare topics I've taken a hard position on. It could be a result of the years spent in the field but also some kind of bias.
> What are the manipulations you’re referring to?
Need to take a step back and mention that in the field of AI there is a great debate between symbolic and non-symbolic approaches. (and after decades spent with AI under symbolic approaches domination we are now in the golden age of non-symbolic AI; with symbolic starting to have a comeback. this podcast can be a good starting point to learn more https://lexfridman.com/gary-marcus/ - although I disagree with GM on many things - and this tweet for learning about symbolic making a comeback https://twitter.com/hardmaru/status/1470847417193209856)
Basically embeddings are "non-symbolic AI" (which is great and this is where their huge potential stems from), but the very way they are generated and then later utlized is completely "symbolic". Which means the the limits of embeddings is defined by the limits of (in this case human written) symbols used to define them. Hope that makes sense.