Comment by lmeyerov
7 hours ago
My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said “from my learned understanding of CompanyX’s internal coding conventions”. When I asked “how do you know about those internal conventions” ChatGPT said “I don’t have access to that internal code, I overstated what I know”. Internal coding conventions are the least of our worries at this point, cat is out of the bag.
You cannot introspect model training by prompting the model. Whatever answer it hallucinated on your query "where do you know that from" has almost certainly very low bearing on reality.
Well how do you know which statement is truthful? These LLMs confidently say they know things that they don't all the time.
For us it's ~easy: we do security investigation CTFs against log dumps / DBs like Splunk. We run the AI CTF as usual.. except without the DB nor internet access. If they still get some right, that's a problem. (And most public CTFs are contaminated now for that reason.)
There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.
I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.
much harder to do in OP's case, matching flavour then referencing that specific companies name when asked how it know to flavour this way? thats astronomically low for randomly selected plausible tokens without some data prior, like that companies codebase.
My own experience is opus being lousy at an extremely niche math task, but it was still easier for me to describe what it needed to do to get code and correct issues in its reasoning/working than to write myself. a minor model number change later and it's nailing everything, despite my opt-out. Its is astronomically unlikley others were working on this also, especially at that level, especially this application.
so, safe to say they _all_ train models on chats, the only difference being if you "opt out" you at least have some defence later when they steal your work and claim it as their models original output.