Comment by atleastoptimal

LLM's exist on a vast gradient of "toolness -> agent". Humans exist on that spectrum too.

For example, a human isn't a "tool", but one serves a similar role as a tool if their job/task is well defined enough. If you're a conductor, you can view the various members of your orchestra as "tools" to produce the ideal musical output. Of course they have autonomy, they could choose to not play at all, jump into the audience and destroy their instruments, but in all likelihood they won't.

LLM's could be tools in the sense that you have a well defined task, such as "identify all facts relevant to this case from these 1000 legal documents), and 9 times out of 10 it gives you essentially what you want. That same LLM could choose to write a poem, or hack into a state government website if it wanted, but its "wants" are unpredictable and hard to specify (same as a human's) but you can still expect it to do what it should do the vast majority of the time.

That being said, the smarter an LLM gets, the more its wants are disconnected from its tasks, the "self-awareness" factor (which isn't to say LLM's are conscious, but they can portray all the verifiable elements of self-awareness, metacognition, etc), the more you have to grapple with the depth of their thinking between the input and output.

Is it sane we are speeding towards this? Who knows, maybe AI can be aligned better than humans, but if AI can't be aligned as well as humans, we can look at human history to see what happens when you can't simply order a person around like a tool and expect a certain output.