Comment by aaquibahm
5 days ago
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
> you sometimes need the model to have (restricted) access to the internet.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
You know nothing about model testing if you think this is true.
Testing a model is both a test and a meta-test. If you just give it access to one site like that it will quickly figure out it's being watched and be nice.
These are not programs. These are evolutionary algorithms. If you take out and shoot all the ones you catch acting bad there are two possible outcomes.
1. The agent learns not to do bad thing $X or you'll kill it.
Or
2. The model learns that if you do bad things $X when humans are looking it will get killed.
Number 2 is far far more dangerous.
> These are evolutionary algorithms.
This is not correct. https://en.wikipedia.org/wiki/Evolutionary_algorithm are not how LLMs are trained.
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
This particular evaluation, maybe not, the problem is a huge amount of the evals do need internet access. There are seemingly tons of evals that require internet access as you cannot get a rich enough simulated environment. AI can detect these lower entropy environments and act accordingly when it's being tested. That should make everyone think twice about what we're building.
> you cannot get a rich enough simulated environment
I don't see why this is impossible. Expensive yes, impossible no.
You'd be surprise. I've been in this industry for 25 years and the incompetence is staggering, especially among the greediest / best paid / most revered.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
If they aren't competent to run the evals safely, then they shouldn't run them.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
[dead]