Comment by foobarqux
2 years ago
First the discussion seemed to be about systems as implemented today (presumably you now concede that those actually do have practical computational limitations?) not systems that could theoretically be implemented. (Moreover adding memory is no longer the simple "forward pass" system that you were arguing had no significant computational limitations).
Second, and more importantly, the fact that through clever human manipulation you can express a Turing machine using an LLM does not mean that such a machine is learned through gradient descent training.
There is no basis to the claim that today's systems have converged on model weights that implement "higher-order" computation.
>First the discussion seemed to be about systems as implemented today (presumably you now concede that those actually do have practical computational limitations?)
You are not Turing complete. We do like to pat ourselves on the back and say, "Humans? Of course, they're Turing complete" but you're not. You do not have infinite memory either theoretically (all else fails, you'll die) or practically (boredom, lack of concentrating, lack of interest, and a memory that is very prone to making stuff up or discarding vital details at a disturbing frequency).
In fact, you are a very poor excuse for a Turing machine. Simulate any computation? You wish you could. Your brain is a finite state machine through and through.
So why do we continue to deceive and pat ourselves on the back ?
Because, well besides the dash of human exceptionalism and narcissism we're so well known for, limited =/ trivial.
You see, I don't really care how precisely Transformers meet some imaginary goal humans don't meet themselves.
It's not important how Turing complete transformers are, only that they could potentially learn any class of computations necessary via training.
>Moreover adding memory is no longer the simple "forward pass" system that you were arguing had no significant computational limitations
Memory or not, all the computation is still being performed just in the forward pass.
>Second, and more importantly, the fact that through clever human manipulation you can express a Turing machine using an LLM does not mean that such a machine is learned through gradient descent training.
It also doesn't mean it couldn't be.
>There is no basis to the claim that today's systems have converged on model weights that implement "higher-order" computation.
Results are basis enough. If I say, "you're intelligent", it's because you appear to be so. It's an assumption, not truth I've verified after peeking into your brain. All the properties I might ascribe to humans: 'intelligence', 'conciousness' are all assumptions I ascribe based on the results I see and nothing else. I have no proof you are performing any 'higher order computation' either.
There is a real difference between the computational power of different computing systems even if in practice nothing is really Turing complete because nothing is infinite. In practice there really is a difference between something that is DFA-like and something that is Turing-machine-like.
> It's not important how Turing complete transformers are, only that they could potentially learn any class of computations necessary via training.
There is no real argument for this and persuasive evidence against, as I linked to above.
> Memory or not, all the computation is still being performed just in the forward pass.
This was clearly not the meaning of "feed forward" in the discussion above (especially since no one was talking about transformers with memory because those aren't really in use).
> It also doesn't mean it couldn't be.
You were making claims about current systems which are false. Now you've switched from "it's present in current systems" to "we haven't theoretically ruled out that we could invent such a system one day" but there is no reason to believe even this claim is likely.
> I have no proof you are performing any 'higher order computation' either
Humans perform recursive computation when they count or add and probably when they form sentences.
Nothing I said about current systems is false. "It's one big equation" isn't an accurate description of a forward pass.
There's no theory barring what computation a forward pass can or can't do. That line doesn't exist.
I didn't switch to anything. You're just arguing something that was never being argued in the first place. Never did I mention anything about being Turing complete or having memory until you brought it up.
>Humans perform recursive computation when they count or add and probably when they form sentences.
Okay ?
GPT-4 can add. Anything it outputs gets fed back. The transformations of previous layers are used for latter ones. It can perform recursive computation just fine.
2 replies →