I'm all for using a council of LLMs, I wrote a tool for it https://github.com/joelio/owl - but you still need to read through PRs yourself, at the very least.
It’s clear that they want this to be true so bad that they’re just not going to do it, and will spend a ton of money on quality gates and mitigation strategies instead of just reading some code.
I have been talking to people who want to autoreview and autoapprove "minor" AI prs. For security especially, I think if the models weren't enough to prevent the issues, they aren't enough to judge what is minor.
[dead]
Multi-model cross-review is important
I'm all for using a council of LLMs, I wrote a tool for it https://github.com/joelio/owl - but you still need to read through PRs yourself, at the very least.
It's not actually, thats just shoving more shit into the shit pipeline.
Humans need to review this stuff yall there's no way around that, apparently to some, very inconvenient reality.
It’s clear that they want this to be true so bad that they’re just not going to do it, and will spend a ton of money on quality gates and mitigation strategies instead of just reading some code.
You can't rely on people spotting the significance of such changes
Tests would have caught it = https://github.com/rhysd/actionlint injection check
also been a huge fan of zizmor (https://github.com/zizmorcore/zizmor) lately, basically: "am I going to footgun myself?"
I have been talking to people who want to autoreview and autoapprove "minor" AI prs. For security especially, I think if the models weren't enough to prevent the issues, they aren't enough to judge what is minor.
^^
Absolutely.
Nothing in the PR jumps out as a red flag. Unless you know how the internals work, I suppose.
Shouldn't anyone reviewing such a PR know how the internals work?
1 reply →
Are you kidding? It's a very obvious case of quote injection. Not some subtle race condition or anything.
1 reply →
> Nothing in the PR jumps out as a red flag.
Made by AI?