Comment by geoelectric

10 years ago

Quality specialist here who worked on testing Gecko platform targets for Mozilla for several years,

For anything halfway complex, you can't reasonably find all the bugs for precisely the same reason you can't reasonably get full path code coverage out of your tests. The possibilities around interactions (this is where bugs mostly live) scale very, very high as more components are involved. A simple example is that ten boolean flags or checkboxes that interact with each other--or, more accurately, possibly interact with each other, if you can't prove they don't--means 2^10 possible states to check. Most things don't just take two values. Add system-level and user-level non-determinism and good luck.

In quality, we handle this via testing by state equivalence, predicting how common a set of interactions will be and prioritizing accordingly, and other shortcut techniques meant to put a finite amount of effort around reducing risk to whatever won't sink your product, but it's almost impossible to reduce to zero. You have to spend a metric crapton of money to do it, which is why A) you mostly only see that kind of goal in stuff like aerospace and B) planes cost an awful lot.

I haven't even gotten to the part where you don't know all the possible defective side effects that could be there and probably wouldn't think to look for everything possible even if you theoretically could. Keep in mind that you're treating the application behavior as inherently untrustworthy, so you can't just assume you know how it'd fail either. For all you know it'll interact with an OS bug in a novel way. There are surprises everywhere.

The sandbox idea is a really good one because it means you can do this once, presumably in a system of controlled complexity architected in a way to be highly testable and highly reviewable. That's your best bet for getting near zero defects: high modularity with strict interfaces and limited interactions all of which are highly deterministic. You definitely won't get that with something as complex as an entire browser platform.

Making the job as small and easy as possible and only doing it once are probably the key here.