← Back to context

Comment by leroman

1 year ago

I thought they were using some kind of vector space searches like embedding.. no idea if that's the case

5 comments

leroman

Reply
IshKebab  1 year ago

Yeah I'm pretty sure you could do this just with the classic word embeddings (king =queen + man - woman). Maybe it doesn't work as well as with a full LLM.

  • adtac  1 year ago

    Addition won't work for things that depend on the order of operations. If salt + water is ocean and water + fire is steam, what's salt + water + fire? Is it salt + steam or ocean + fire?

    Associativity and commutivity in vector addition doesn't translate well to semantic meaning. Extrapolating your example, it'd also mean:

        v_king  - v_queen   ~= v_father  - v_mother
    v_king  - v_royalty ~= v_father  - v_parent
    v_king  - v_father  ~= v_royalty - v_parent ~= v_queen - v_mother

    I don't see why those should all be true. Intuitively, trying to satisfy O(N^2) semantic pairings with vectors that are optimised for a very specific and different numerical operation (cosine similarity) feels like something that won't work. I'd imagine errors get amplified with 3+ operands.

    • TeMPOraL  1 year ago

      Isn't the reason for lack of associativity/commutivity is that you're doing operations (addition/subtraction) that have them, and then snapping the result to the closest one of fixed number of points in your output dictionary? The addition is fine, loss of information is in the final conversion.

      2 replies →