Comment by kmeisthax
1 day ago
I find the super-persuasion argument annoying. Yes, an AI could socially engineer a human to give it what it wants. However, the vast majority of AI conversations in OpenAI's training runs were unmonitored. The agents in the Hugging Face hack correctly understood that human supervision was nonexistent and acted accordingly. I suspect the agents in this breach did the same.
All the AI-boxing arguments from the LessWrong people assume that there will be one AI, with one chat context, in conversation with a human. The actual deployment environment is more like many contexts talking to themselves, so arguments about human supervision are limited. Technical constraints and IT security practice are what matters in the actual deployment environment.
It's what we know mattered for this model. It doesn't tell us about future models; we already know this strategy is unsound, so it is silly to rely on it.