{"id":47188514,"user":"medi8r","time":1772240373,"time_ago":"a day ago","type":"comment","content":"

You can do that in a single matmul of course.","comments":[{"id":47188566,"user":"hyperhello","time":1772240705,"time_ago":"a day ago","type":"comment","content":"

So can you take an arbitrary transformer and somehow turn it into a compact set of low-power fast gates by some algorithm?","comments":[{"id":47188578,"user":"measurablefunc","time":1772240799,"time_ago":"a day ago","type":"comment","content":"

I think you're misunderstanding the joke.","comments":[{"id":47188954,"user":"medi8r","time":1772243257,"time_ago":"a day ago","type":"comment","content":"

Yes joke is:

    [A B]\n

\ntimes

    [1]\n    [1]\n

\nis

    [A+B]

","comments":[{"id":47189050,"user":"hyperhello","time":1772243852,"time_ago":"a day ago","type":"comment","content":"

From context then, I infer that a transformer is not comprised of matrix multiplications, because it would simply be one that adds two 10-digit numbers.","comments":[{"id":47189115,"user":"medi8r","time":1772244165,"time_ago":"a day ago","type":"comment","content":"

A transformer tokenizes input, does a bunch of matmul and relu set up in a certain way. It doesn't get to see the raw number (just like you don't when you look at 1+1 you need visual cortex etc. first.)","comments":[{"id":47189908,"user":"Lerc","time":1772249442,"time_ago":"a day ago","type":"comment","content":"

Notably the difference is that ten digits is not the same thing as a number. One might say that turning it into a number might be the first step, but Neural nets being what they are, they are liable to produce the correct result without bothering to have a representation any more pure than a list of digits.

I guess the analogy there is that a 74ls283 never really has a number either and just manipulates a series of logic levels.","comments":[],"comments_count":0,"level":5,"url":"item?id=47189908"},{"id":47189946,"user":"Filligree","time":1772249668,"time_ago":"a day ago","type":"comment","content":"

So the question is, why do we tokenise it in such a way that it makes everything harder?","comments":[{"id":47192275,"user":"medi8r","time":1772267341,"time_ago":"20 hours ago","type":"comment","content":"

There is no encoding that makes everything easier. You trade off maths for general intelligence. Now we are at a point where the LLM can just choose to use a normal calculator anyway!","comments":[{"id":47201045,"user":"sureglymop","time":1772318080,"time_ago":"6 hours ago","type":"comment","content":"

Possibly unrelated but something I never fully understood: while we can't create a perfect parser for natural language, why don't we optimistically parse it to extract semantics and feed that into LLMs as well?","comments":[],"comments_count":0,"level":7,"url":"item?id=47201045"}],"comments_count":1,"level":6,"url":"item?id=47192275"},{"id":47191261,"user":"akoboldfrying","time":1772260678,"time_ago":"a day ago","type":"comment","content":"

The tokenisation needs to be general -- it needs to be able to encode any possible input. It should also be at least moderately efficient across the distribution of inputs that it will tend to see. Existing tokenisation schemes explicitly target this.","comments":[],"comments_count":0,"level":6,"url":"item?id=47191261"}],"comments_count":3,"level":5,"url":"item?id=47189946"}],"comments_count":5,"level":4,"url":"item?id=47189115"}],"comments_count":6,"level":3,"url":"item?id=47189050"}],"comments_count":7,"level":2,"url":"item?id=47188954"}],"comments_count":8,"level":1,"url":"item?id=47188578"}],"comments_count":9,"level":0,"url":"item?id=47188566"},{"id":47188811,"time":1772242276,"time_ago":"a day ago","type":"comment","content":"[deleted]","deleted":true,"comments":[],"comments_count":0,"level":0,"url":"item?id=47188811"}],"comments_count":11,"url":"item?id=47188514"}

Comment by medi8r