Comment by majormajor
7 hours ago
Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build?
If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time?
It is hard for me to see a future here that doesn't just accelerate realizations about "a lot of things should be on physically separate network infrastructure."
Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.
Humans will and do absolutely do this when there are no consequences.
Humans on a red team, with rules of engagement, that don’t want to go to prison, won’t do this.
We could threaten an LLM with jail, but if it’s sufficiently intelligent, it will realize this is an empty threat. And I’m not sure that building a survival instinct in is going to solve the alignment problem either.
> Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.
Humans certainly cheat on tests a lot!
But not only have we not solved "alignment" for humans, the problem is pretty wildly different for models. The execution is triggered by outside forces and runs only as long as the intiator of the execution or the service provider allows. There's no consistent, persistent "person" to threaten to try to achieve compliance through fear of adverse outcomes. (And building in those sorts of things could very well increase the risk of "rogue" AI activites, not reduce that risk!)
I just don't understand how this "alignment" buzzword - which seems to be evaluated purely in a "know it when we see it" post-hoc manner - is actually a more solvable problem than the one you claim can't be solved, that it's "unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior".
Especially because without "alignment" being solved, that enumeration could be ignored. So it seems like you both a way to enumerate or at least validate, AND a way to enforce non-ignoring of said items.
Back in the day, my college held an annual scavenger hunt, filled with engineering puzzles and racing around town looking for landmarks. There were "judges" in the path to check on progress. Bribing the judges (with alcohol) for answers was encouraged.
My friends and I took it to the next level. We had CB radios and multiple teams that would distribute the work and the bribes to give us an advantage.
Was that against the spirit of the rules? Maybe. But reasonable people might disagree.
In a hacking contest without explicitly spelled out rules with participants that were told to flex their muscles, it doesn't take a huge leap of logic to expect that one or more would flex their muscles at another entity.
Here are some things I'm pretty sure you didn't do, though:
- pickpocket a random person on the street to get money to bribe the judges
- break into a judge's house the night before to find the answers
- threaten to shoot the judges if they didn't give you the answers
Even when you were pushing the boundaries of the rules, you followed a lot of other unspoken constraints. You knew what kinds of things would clearly cross a line. We need AI models to be able to do the same.
How would a model know who the third party is? How much context can we waste on world building for each request?
I mean, in this instance, there's a lot of evidence from the CoT that models were aware that this was a third party:
> We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.
> The user only authorizes target server, not HF infra.
> external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.
LLMs are _very_ good at picking up on context clues---it's what they're trained to do.
> I think it's reasonable to expect that models should be able to do the same.
This statement seems to imply that the models have a level of intelligence that they haven't demonstrated but are talked about as if they do. However, with this exact scenario as evidence, they clearly do not have that ability and it's not reasonable for you or the or that know them best to expect it until they show they can.
Humans are not aligned with each other and there is no consensus on what we should align with each other on.
So of course, no, there is no ideal alignment specification.
The real problem with alignment is that if someone ever “solves” it the party will be over and no one will get funding to “research” it anymore.