Comment by miguelspizza

The more I use the AIs the more I feel like I did at the end of reading blindsight. The things are undoubtedly “intelligent” by any practical definition of the word, but they are not aware.

This is also why I advocate against using AI as a writing partner. No matter the argument you lay out, a fresh context window will always have the “a few good things and a few bad things” feedback. There is no higher order opinion to align with.