Comment by sigbottle

Again, I will posit the hypothesis that it's a learned behavior from training.

Distinctions, you generally "only pay for" in computational cost, by needing to search twice over an axis you may not need to split.

Similarities, if you wrongly assume two things are similar, means you're just wrong.

Of course, we know from computer science that doing more computation isn't free either.

I find myself often being more and more pedantic the more I want correctness - but of course this comes with the tradeoff of losing the high level abstract picture.

Saying what you're not going to do is also good design hygiene.

I will say that I'm annoyed by this behavior too. It feels like the models are writing their state of mind directly to output that should be clean. Often times, I will push back, and then it will... do the correction, and write the push back into the damn output. "Claude, I want burgers, not fries". The button text now changes to "Fries (NOT BURGERS)". Like, what?

Distinctions are powerful local reasoning tools, but a component of "real" reasoning is synthesis. Which they clearly can do sometimes - but not every time and not even remotely a probable amount of times.