Comment by hn_throwaway_99

Thanks for your correction. This makes me think that how Gemini arrived at this answer is that it mashed together "heating first" with "no heating but with citric acid first" articles, but left out the (critically important) citric acid part.

I think this "failure mode" really highlights how LLMs aren't "thinking", but just mashing up statistically probable tokens. For example, there was an HN article recently about how law-focused LLMs made tons of mistakes. A big reason for this is that the law itself is filled with text that is contradictory: laws get passed that are then found unconstitutional, some legal decisions are overturned by higher courts, etc. When you're just "mashing this text together", which is basically what LLMs do, it doesn't really know which piece of text is now controlling in the legal sense.