Comment by witx
13 hours ago
What, no they're not. You still need to analyze them to understand they are false positives. It's time wasted
13 hours ago
What, no they're not. You still need to analyze them to understand they are false positives. It's time wasted
Agree, it's something that will eventually teach your developers to ignore points raised as it's mostly garbage.
Finding problems is optimizing for the customer. Avoiding false positives is optimizing for the developer. Which is right depends on your org's culture.
If I flag every line in your PR as a potential security bug then I have 100% recall.
Obviously you need a mixture of high recall and low false positive rate. If 7/8 flagged items are fine its much more likely people will ignore the warnings, much like they would any security tool with a 90% false positive rate. That is not optimized for the customer.
The ideal is finding all the problems without getting any false positives, but the reality is that you can't often have that. An org's engineering culture should be designed to fix problems with systems. If you're seeing an 87.5% false positive rate that should be seen as another engineering problem to fix. However, that's a separate issue to whether or not you accept false positives in a system designed to find problems.
Presenting it as either a system that misses real problems or a system that has a huge number of false positives is a false dilemma. You can have a system that's designed to find all the problems and then optimize it to reduce the false positives. If you can't reduce the number then you optimize to identify false positives as fast as possible. Just ignoring the identified problems on the assumption that they're false is giant red flag and a signal that the org has a very a broken engineering culture (but, as you say, that's quite common.)
Yep. Similarly - you can predict with 99.9% accuracy if a Volcano will erupt today by using a rock that has "No" written on it.
> If I flag every line in your PR as a potential security bug then I have 100% recall.
No. A code review isn't about "flagging a line of code", it's about identifying an issue or a risk. If a 10-line PR has one issue and you leave a comment on every single character, if you still miss the issue you have 0% recall.