Comment by user43928
1 day ago
It's not stupid if it works.
Concretely, I would like to see it benchmarked:
Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?
I wouldn't be surprised if it's zero times.
There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.
Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.
That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.
> Have Opus 5.5 implement the authentication
“Authentication” is extremely far from being the only relevant characteristic of security. That in fact is the whole problem with security: it’s a pervasive concern that has to be considered throughout a system, at every level.
The way the bots fundamentally work means that we can never trust their output. We have ample cases at work where bots catch human mistakes and humans catch bot mistakes. The important aspect here is not completely handing things over to the bots.
> I wouldn't be surprised if it's zero times.
Statements like this concern me.
If you're so concerned you can create such a benchmark and show us all how wrong my guess is.
Instead I mostly read such opinions based on active imagination and no concrete experience with using AI for such work in a setup as described.
I suspect you would disregard or find fault in any benchmark I or others provide you.
Alternatively, if you want to see a benchmark so badly, you can create such a benchmark for your guess. I would imagine you would have more faith in the results of your own work.
6 replies →