Comment by weinzierl
7 days ago
"I thoughts LLMs were glorified Markov chain generators"
"the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did. I was shocked."
These two views are by no means mutually exclusive. I find LLMs extremely useful and still believe they are glorified Markov generators.
The take away should be that that is all you need and humans likely are nothing more than that.
I suppose it's all a continuum and we can each have different opinions on what the threshold for "glorified markov generator" is.
But there have been many cases in my experience where the LLM could not possibly have been simply pattern-matching to something it had seen before. It really did "understand" the meaning of the code by any definition that makes sense to me.
> It really did "understand" the meaning of the code by any definition that makes sense to me.
I find it dangerous to say it "understands". People are fast to say it "is sentient by any definition that makes sense to them".
Also, would we say that a compiler "understands" the meaning of the code?
> humans likely are nothing more than that
Relevant post: https://news.ycombinator.com/item?id=44089156
> I find LLMs extremely useful and still believe they are glorified Markov generators.
Then you should be able to make a markov chain generator without deep neural nets, and it should be on the same level of performance as current LLMs.
But we both know you can't.
You can, but it will require far more memory than any computer has.
The way the input doesn't match the output should imply that it's not just statistics.
As soon as compression happens, optimization happens which can lead to rules/learning of principles which got feed by statistics.
That's "just" more statistics though.
Are you good in math definitions or is this an opinion?
For me a compressed model learning rules through statistics is not statistics anymore.
Physic rules are not statistics.
3 replies →