Comment by ffsm8

there is an established term for this behaviour: hallucination.

lets me rephrase what i said before as on retrospection the point i was trying to make didnt come across:

Asking the model such as question is a pointless endeavor. It will pretty much always answer in a plausible sounding manner to whatever question you gave it.

Its like another top HN story - an opus5 parody site right now, https://opusfived.dev/

pretty much the whole dialog is plausible given the prompts the user is required to make to illustrate the point the author of the site wanted to make... but the thing thats ironic about the site is that ... if youre prompting the model badly, it outputs garbage. though i'm pretty sure the author didn't want to make that point, to me its a perfect illustration of this.

And the same applies to asking the model to explain themselves and the related question. Asking for a source of the term is fine, and it will either respond with one and its cleared up -- or not and you know its a hallucination. nothing special about it, as they constantly hallucinating. its usually just self-correcting during the agentic loop.