Comment by lmeyerov

For us it's ~easy: we do security investigation CTFs against log dumps / DBs like Splunk. We run the AI CTF as usual.. except without the DB nor internet access. If they still get some right, that's a problem. (And most public CTFs are contaminated now for that reason.)

There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.

I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.