← Back to context

Comment by strange_quark

17 hours ago

Wasn't it already confirmed that small open-weight models were able to detect most of the same headline vulns as mythos? How is this any different?

7 comments

strange_quark

Reply
stanfordkid  17 hours ago

No, they are able to detect errors when pointed at them but they have a lot of false positives... making them functionally useless for a large unknown codebase. They also can't build and run an exploit post-identification. Mythos can find vulnerabilities (purportedly) and actually validate them by building and running exploits. This makes it functional and usable for hacking.

  • adrian_b  4 hours ago

    The only significant difference between Mythos and the older open-weights models was that Mythos found all the bugs alone, while with the older models you had to run many of them in order to find all bugs, because each model found only a part of the bugs.

    For the open weights models, we know the exact prompts that have been used to find the bugs. While the prompts had to be rather specific, a good bug-finding harness should be able to generate such prompts automatically, i.e. by running repeatedly a model while requesting to find various classes of bugs.

    For Mythos, we do not know what prompts have been used, but Anthropic has admitted that the process was nothing like asking "find the bugs in this project". They have also run Mythos many times on each source file, starting with more generic prompts in order to identify whether a source file is likely to have bugs, and then following with more and more specific prompts, until eventually it became likely that a certain kind of bug exists, when Mythos was run one last time with a prompt that required the confirmation that the bug exists and the possible generation of an exploit or patch.

    So Mythos must also be pointed to an error. Using it naively will not provide any results like those reported.

    There is no doubt that both Mythos and GPT 5.5 are superior to older models, because you can use a single model and hope to have an adequate bug coverage. But the difference between them and older models has been exaggerated. If you run older models on your own hardware, you can afford to run many models many times on each file. A serious bug searching with Mythos or GPT 5.5 is likely to be very expensive, while likely to provide the same results in most cases.

  • dlahoda  14 hours ago

    i casually asked gemini and codex 200usd subs to find and verify bugs for weeks. it did wrote tests, injected mutations, verified fixes. just promts.

    also i had to proxy remote mainnet with localhost to force them to do penetration and dos testing.

    mythos is nothing new.