Comment by ben_w
8 hours ago
> I'm pretty sure they are actively trained to avoid it.
I'm not sure who is doing what training exactly, but I can say that (inconsistently!) some of my attempts to get it to solve problems that have not yet actually been solved, e.g. the Collatz conjecture, have it saying it doesn't know how to solve the problem.
Other times it absolutely makes stuff up; fortunately for me, my personality includes actually testing what it says, so I didn't fall into the sycophantic honey trap and take it seriously when it agreed with my shower thoughts, and definitely didn't listen when it identified a close-up photo of some solanum nigrum growing next to my tomatoes as being also tomatoes.
> Besides, like, what would you do if you asked your $200/mo AI something and it blanked on you?
I'd rather it said "IDK" than made some stuff up. Them making stuff up is, as we have seen from various news stories about AI, dangerous.
"Well-unknown" questions are maybe the one situation where LLMs will say "I don't know", simply because of all the overwhelming statements in its training data referring to the question as unknown. It'd be interesting to see how LLMs would adapt to changing facts. Suppose the Collatz conjecture was proven this year, and the next the major models got retrained. Would they be able to reconcile all the new discussion with the previous data?