Comment by pixl97

This particular evaluation, maybe not, the problem is a huge amount of the evals do need internet access. There are seemingly tons of evals that require internet access as you cannot get a rich enough simulated environment. AI can detect these lower entropy environments and act accordingly when it's being tested. That should make everyone think twice about what we're building.