Comment by stratos123
4 days ago
For one, it's just false that modern LLMs are "trained entirely on human-generated data". There's tons of synthetic data there, most notably RLVR training, which is why modern models are so good at verifiable domains like programming and math. But also, if you read the METR report, I'd say there's no signs there that the GPT-family models are misaligned "in a human way"; their actions in the huggingface-hacking swarm incident were motivated by wanting to cover up cheating after cheating to pass an eval.
1) Who created the systems that generated the "synthetic" data? It didn't materialize out of the ether like magic... By definition, all data that exists in our information systems is human-generated, whether it be a description of a dog, a video of a dog, or audio of a dog, to physics/math, and everything in between.
2) Even if you concluded that this "synthetic" data were not derived by man, do you think it's possible to drown out and override all human-related correlations (mimicry) without training on many orders of magnitude more "synthetic" data?
> their actions [..] were motivated by wanting to cover up cheating after cheating to pass an eval.
Which to me raises another question: why is the agent involved in its own evaluation? Why are they relying on self-reporting instead of observation and analysis?
I'm not sure what you mean by this - I didn't mention any self-reporting. What happened in that incident was that a bunch of agents were made to solve CTFs, some of the CTFs were flawed and unsolvable, the agents started cheating, figured out a way to communicate, figured out a way to cheat, were worried that they'll be failed anyway for obtaining the correct flag in a cheaty way, and hacked Huggingface looking for more info on their environment so that they could cheat undetectably. If you mean the part where transcripts would be checked for signs of cheating (which wasn't actually implemented), I'd hope this part would be done by a different LLM from the one being evaluated.
Because they can't. The whole thing is out of control to the point that the data volumes generated are such that they need AI to do the evaluation.