Comment by theptip
5 days ago
If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up? I feel your ”with the right harness” may be doing too much work here.
Certainly no existing harness today, nor a year ago, could achieve a fully autonomous e2e own with a ‘25 class open model. Even with Opus 3 series I don’t buy it!
> If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up
There are multiple AI network pentesting and redteaming startups that have been on the market for at least 3-4 years now and have conducted similar actions.
Horizon3 and Pentera off the top of my head, but I remember Crowdstrike, SentinelOne, and Wiz had similar capabilities on roadmap around 18 months ago (and in Wiz's case GAed).
Well… only half of your premise is actually verifiable. We know that no one wrote it up, but we don’t know that no one did. What I can say… even the smaller (like 35B) “abliterated” Qwen models have impressive red-team capabilities for what they are, even without a harness. Just manually giving them a set of facts and asking “What next?” will get you reasonable next steps for trying to find vulnerabilities in webapps. You then manually keep track of the facts you learn and add appropriate detail (blog -> Wordpress -> Wordpress x.y.z with these plugins:…) and just feed it in a loop. Could pretty easily put together a minimal harness to do that.
And that’s all just “from memory” without something like RAG or web search access…
If anyone wants to run the guardrail-free Qwen models, here’s a link to the download plus a script to run it: https://news.ycombinator.com/item?id=49002475
Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.
The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).
There are whole companies premised on this. No existing harness you personally can go download, sure. And you're setting an artificially high bar when you talk about Mythos; I'm not saying you could just plug a 2025 open weights model into OpenCode (or whatever it was in 2025) the way you can apparently plug Mythos into standard Claude Code and have it cook.
Isn't the answer to this obvious? Because you, I, and anyone else who isn't working for a big AI company cannot disable the guardrails, which is what made this possible. It doesn't mean old models weren't capable of it.
I don’t buy it. We are hearing a lot about Mythos capabilities now. If they existed a year ago they would have been trumpeted then.
Look, a lot of energy from third parties goes into demonstrating the maximally bad thing a model can do (and the standard response around here is always to throw shade on frontier capabilities).
It has nothing to do with the guardrails; Pliny jailbreaks those within hours. It has everything to do with task horizon coherence and overall IQ.
It’s simply revisionist IMO to claim this stuff was latent all along.
This wasn't just capable last year, it was capable prior to ChatGPT existing entirely through specialized automation harnesses that predate the terminology of a "harness" for models. This is something you could have done in 2018 if you had a spare couple of billion dollars of compute to waste. It was a subject discussed at security conferences even before that.
Automated dynamic exploit chains and credential discovery has been something every red team worth their salt does at a lesser scale for 20 years, why would you ever think that the capability didn't exist until it only cost a few thousand dollars to do?
Existed does not mean was easy, or was cheap, or was thought of as reasonable to attempt.
Also, IQ is not a term that applies to language models, and the term you are looking for is "Long Horizon Coherence".