Comment by Closi

Because humans always say 'bread' if you ask them what you put in a toaster.

And humans will always deduce that you should switch doors if you are in a hypothetical gameshow and they show you a horse behind one of the doors.

(All I mean is - an example of an LLM answering illogically is not proof that LLMs can't really think logically, as you can equally find examples of humans answering illogically and also find examples of novel questions that LLMs get right)