Comment by mindslight

It's not "AI" - its an LLM that is doing the equivalent of a college freshman padding out a paper for length. Confident, verbose, polished, but ultimately based on little hard reasoning - aka bullshitting. When it's wrong, and if you notice it and call it out, it will happily apologize and "correct" itself with the same well-written prose while making another mistake (or even the same exact one). LLMs certainly have utility, but it's more as generating inputs to some verifying processes rather than as a standalone oracle that competently answers questions.