Comment by Borealid
7 hours ago
> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|
"You can't know what the impact of your action will be if it doesn't change much"
> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large
"You can't necessarily change much"
> (or that f_actual(X) saturates its codomain)
"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.
> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
> "You can't know what the impact of your action will be if it doesn't change much"
Incorrect translation. No "if" clause need apply.
> "People can't do arbitrary things"
Also incorrect. f_actual(X) is a product of both "how much people can do" and "how much you can control/persuade them." I would model each of these with their own precise terms and relate them back to f_actual(X), and we could continue like this indefinitely, but from now on I'll just give you my reply in English: Not only does The Atomic Boy Scout not prove that a random person can bring about mass destruction (collecting random radiological material and contaminating the local area is not developing a nuke, nor is it even an effective dirty bomb, because it was neither very dirty nor a bomb; the safeguards were not lack of knowledge, either), but it still remains to be demonstrated that individuals will be radicalized by AI just by virtue of the AI being, like, super-duper smart.
> "Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
What domain of input do you propose, then? LLMs as rational agents have no agency but for what tokens they generate. What we hook them up to, and what they could plausibly do with what we hook them up to, has been left unsaid.
Denying the premise, which is that an intelligent being is one who can predict the outcome of actions with sufficient precision to influence the future, makes debate pointless.
I literally granted you that premise:
> even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.