Comment by starbugs

> What does self referential have to do with anything?

It means that the definition of "thought" from Webster as "an individual act or product of thinking" is referring to the word being defined (thought -> thinking) and thus is self-referential. I said in my prior response already that if you refer to the input of the model being a "product of thinking", then I agree, but that doesn't give the model an ability to think. It just means that its input has been thought up by humans.

> When a computer is busy or loading, they might say "it's thinking."

Which I hope was never meant to be a serious claim that a computer would really be thinking in those cases.

> You could certainly ask this model to write up a plan for something.

This is not the same thing as planning. Because it's an LLM, if you ask it to write up a plan, it will do its thing and predict the next series of words most probable based on its training corpus. This is not the same as actively planning something with an intention of achieving a goal. It's basically reciting plans that exist in its training set adapted to the prompt, which can look convincing to a certain degree if you are lucky.

> Whether you like it or not, these LLMs do have some limited ability to reason.

While this is an ongoing discussion, there are various papers that make good attempts at proving the opposite. If you think about it, LLMs (before the trick applied in the o1 model) cannot have any reasoning ability since the processing time for each token is constant. Whether adding more internal "reasoning" tokens is going to change anything about this, I am not sure anyone can say for sure at the moment since the model is not open to inspection, but I think there are many pointers suggesting it's rather improbable. The most prominent being the fact that LLMs come with a > 0 chance of the next word predicted being wrong, thus real reasoning is not possible since there is no way to reliably check for errors (hallucination). Did you ever get "I don't know." as a response from an LLM? May that be because it cannot reason and instead just predicts the next word based on probabilities inferred from the training corpus (which for obvious reasons doesn't include what the model doesn't "know" and reasoning would be required to infer the fact that it doesn't know something)?

> I'm not here to praise chatbots or anything, but I also don't have a blind hatred for the technology, nor do I immediately reject everything labeled as "AI".

I hope I didn't come across as having "blind hatred" for anything. I think it's important to understand what transformer based LLMs are actually capable of and what they are not. Anthropomorphizing technology is in my estimation a slippery slope. Calling an LLM a "being", "thinking" or "reasoning" are only some examples of what "sales optimizing" anthropomorphization could look like. This comes not only with the danger of you investing into the wrong thing, but also of making wrong decisions that could have significant consequences for your future career and life in general. Last but not least, it might be detrimental to the development of future useful AI (as in "improving our lives") since it may lead to deciders in politics drawing the wrong conclusions in terms of regulation and so on.