Comment by kelseyfrog

None of our first encounters with "AGI" will be with a paperclip maximizer. It will sneak up on you in a moment of surprise. It will be the unmistakable jolt of recognition, an inner voice whispering, "Wait...this thing is real. I’m talking to something that knows me."

I’m not saying user experiences are AGI; I'm saying they functionally instantiate the social and psychological effects we worry about from AGI. If people start treating LLMs as conscious agents, giving them trust, authority, or control, before the systems are actually capable, then the social consequences precede the technical threshold. On the divinitory scale between Ouija boards and ChatGPT, Ouija matters less because their effects are limited while AIs, by design, are deeply persuasive, scalable, and integrated into decision pipelines. Sometimes the category error is upstream. The risk is things that seem like AGI to enough people become as dangerous as AGI. That may happen well before AGI arrives.

The danger isn't that we build a AI that surpases some AGI performance threshold and it goes rogue. The danger is that we build systems that exploit (or are exploited by) the bugs in human cognition. If the alignment community doesn't study these, the market will weaponize them. We need to widen the front lines on alignment research to include these cases. Without changes, the trajectory we're on means there will be more.