Comment by maevyn11

Yea I think if you read the actual design of the test they are presenting as evidence it shows that what these small models are doing is not the same as what Mythos did. They isolated the vulnerable code down to the vulnerable subset of the function and provided hints in the prompt about all of the key contextual factors that matter to finding the vulnerability. That makes the problem significantly easier.

I realize they are trying to prove that an agentic harness running small models can ultimately achieve the same thing as what Mythos did, but they are handwaving away the steps it takes to construct the context Mythos handled in model and using a misleading test result to prove small models can handle the key step.

Poor evidence of a premise that logically wouldn't even be proven if the their evidence was valid. If they could find these types of vulnerabilities with the same effectiveness they would have done it already.