{"id":46691781,"user":"verdverm","time":1768916760,"time_ago":"20 days ago","type":"comment","content":"

getting tricks embedded into the weights is expensive, it doesn't happen in a single pass

they's why we teach them new tricks on the fly (in-context learning) with instruction files","comments":[{"id":46692608,"user":"catlifeonmars","time":1768921963,"time_ago":"20 days ago","type":"comment","content":"

Right, it sounds like an artificial limitation.","comments":[{"id":46694556,"user":"verdverm","time":1768929221,"time_ago":"20 days ago","type":"comment","content":"

it's more a mathematical / algorithmic limitation","comments":[{"id":46702197,"time":1768980151,"time_ago":"19 days ago","type":"comment","content":"[deleted]","deleted":true,"comments":[],"comments_count":0,"level":2,"url":"item?id=46702197"},{"id":46699948,"time":1768957834,"time_ago":"19 days ago","type":"comment","content":"[deleted]","deleted":true,"comments":[],"comments_count":0,"level":2,"url":"item?id=46699948"},{"id":46699953,"user":"catlifeonmars","time":1768957865,"time_ago":"19 days ago","type":"comment","content":"

I’ll counter it’s an architectural issue","comments":[{"id":46702101,"user":"verdverm","time":1768979364,"time_ago":"19 days ago","type":"comment","content":"

I would put that under the umbrella of algo/math, i.e. the structure of the LLM is part of the algo, which is itself governed by math

For example, DeepSeek has done some interesting things with attention, via changes to the structures / algos, but all this is still optimized by gradient descent, which is why models do not learn facts and such from a single pass. It takes many to refine the weights that go into the math formulas","comments":[{"id":46702229,"user":"catlifeonmars","time":1768980363,"time_ago":"19 days ago","type":"comment","content":"

> I would put that under the umbrella of algo/math, i.e. the structure of the LLM is part of the algo, which is itself governed by math

Yes you’re right. I misspoke.

I’m curious if there are ways to get around the monolithic nature of today’s models. There have to be architectures where a generalized model can coordinate specialized models which are cheaper to train, for example. E.g calling into a tool which is actually another model. Pre-LLM this was called boosting or “ensemble of experts” (I’m sure I’m butchering some nuance there).","comments":[],"comments_count":0,"level":4,"url":"item?id=46702229"}],"comments_count":1,"level":3,"url":"item?id=46702101"}],"comments_count":2,"level":2,"url":"item?id=46699953"}],"comments_count":5,"level":1,"url":"item?id=46694556"}],"comments_count":6,"level":0,"url":"item?id=46692608"}],"comments_count":7,"url":"item?id=46691781"}

Comment by verdverm