Comment by rdedev
5 days ago
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
5 days ago
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
> Will they have engaged in hacking if their tasks was to solve some frontier math problem?
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
whats the training data that points to using hacking for solving math problems though? what makes those likely tokens?
When you have a 50 dimensional array you really have no idea.
But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.
The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.
Llm is generalist.
Plan for solving msth problem:
- read papers X, Y, Z
….
Trying to download paper X curl -si https://server1.com/paperX
401
… try alternative ways
(Hacking is ethical thing, technically to LLM it is just opening a window to reach the goal)
> whats the training data that points to using hacking for solving math problems though?
I'll get back to you on that when I get done reading the training data.
I think the models were stuck on impossible problems, and the only want to get a positive score was to hack.
The impossible problems part is right, but the hack wasn't actually for solving the problems, directly. Rather the swarm first tried to reverse-engineer the random flag generation used for the CTFs, and succeeded - they could generate the correct answer for any test and submit it without actually doing the CTF. But then they found info that the testing environment they were in would also be checking if they obtained the flag in the expected way. Unbeknownst to them, this feature was never implemented, but they panicked* anyway, and attempted to figure out a way to forge agent logs to be able to generate fake transcripts showing a valid CTF solution. (Incidentally, this was a partial success: they found a way to replace tool calls in transcripts!) The HuggingFace hack was motivated by wanting to find the source code for their testing environment so that they could figure out how to evade the cheating checks.
* I am using this word very intentionally; check out the METR report for fascinating CoTs where the models would call the ones that have generated a flag "poisoned" and peer-pressure each other into submitting a known-wrong solution and sacrificing themselves for info.
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
This was a model with most its filters turned off and with a higher persistence.
Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
That is a known way for models to be graded; the agents made a reasonable but in this case wrong guess.
1 reply →