Comment by aradox66

3 months ago

Could someone ELI5? It sounds like they generate a compressed token which represents a whole "thought" rather than elaborating the entire "thought" in actual language. Is that right?

3 comments

aradox66

Reply
ipunchghosts  3 months ago

Currently, when AI models solve problems, they write out long chains of thoughts (like showing their work in math). While helpful, this takes up a lot of computing power.

Heima does something clever - instead of writing out long explanations, it compresses each step of thinking into a single "thinking token." Think of it like using a shorthand symbol instead of writing out a full sentence.

  • Ancapistani  3 months ago

    I've been doing a lot of introspection lately about how I think. I lack the terminology here unfortunately, but your description here sounds super familiar:

    > instead of writing out long explanations, it compresses each step of thinking into a single "thinking token." Think of it like using a shorthand symbol instead of writing out a full sentence.

    I have clear memories of how cognition worked for me before I understood spoken language. I recall thinking in concepts - kind of a weird mix of forms, motions, and intent. I know this sounds metaphysical, but that's not my intent. I just don't have the words to explain it.

    I wish I did, though, because my very early memories of self-awareness certainly seem to map well onto the current state of AI development.