Comment by ainch
9 hours ago
This is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests.
I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are few consequences for hacking.
Just a wild guess - perhaps coding, being one of the things these models are most heavily trained for, is such a strong predictor that it typically keeps it on track?
I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.
Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?
> download source for bison, build it, find and download an ANSI C grammar, build the parser, etc
Couldn’t have trained a better token-burner if they tried.
Human hackers would be put into jail. LLM hackers would be rewarded.
Why do you find it hard to reconcile?
It seems fair to guess you aren’t building your own harnesses, running multiple experiments with achievable win conditions, while providing oversight over model behavior.
Scale alone suggests your experience won’t match.
In the future, When everyone and their uncle is launching swarms to solve impossible challenges, at that point we can expect proliferation of this scenario all over the place.