Comment by rho_soul_kg_m3

And I didn't say it does. Intelligence is not necessarily deterministic, and being random is not the problem with LLMs. The problem is that they are not intelligent: they statistically mimic reasoning and logic, which still could have been acceptable except that they don't generalize well and have double-digit (at best single-digit) error rate percentages.

They also have the worst possible failure mode imaginable: Producing erroneous output that looks perfectly fine and expertly-crafted.

Imagine a food synthesizer machine. You press a button. 80% of the time you get a chicken sandwich, 20% of the time it beeps an error. That's OK. With the LLM version of that, 80% of the time you get a sandwich, 20% of the time you get what looks like a perfect sandwich except that it contains bits of plastic and metal, and you have to start eating it to find the pieces.

"You're absolutely right! Food shouldn't contain bits of plastic. Let me synthesize that again."