Comment by faurroar

I don't think it's surprising, per say, but that's a consequence of the fact that I don't believe there is some sort of magic threshold at which a system becomes agential.

Like I don't necessarily disagree with any of your framing. The thrust of the alignment problem, as I see it, is that there is an intrinsic problem of aligning the goals of two distinct systems that poses catastrophic risks precisely when one of the systems is significantly more capable (in some sense or other, maybe not in a general/absolute sense) than the other.