← Back to context

Comment by mrits

17 hours ago

Author should ask AI to write a small app with 100% code coverage that breaks in every path except what is covered in the tests.

8 comments

mrits

Reply
thih9  17 hours ago

Example output if anyone else is curious:

    def fragile(x):
        lst = [None]
        lst[x - 42]
        return "ok"
    
    def test_fragile():
        assert fragile(42) == "ok"

  • andrekandre  13 hours ago

    this doesn't seem like a very useful test...? i'm more interested in the failure modes when input != 42, what happens when i pass NaN to that etc...

    jmo, but tests should be a chance to improve the implementation of functions not just one-off "write and forget" confirmations of the happy path only... automating all that just short-circuits that whole process... but maybe i'm missing something.

sgk284  17 hours ago

I never claim that 100% coverage has anything to do with code breaking. The only claim made is that anything less than 100% does guarantee that some piece of code is not automatically exercised, which we don't allow.

It's a footnote on the post, but I expand on this with:

  100% coverage is actually the minimum bar we set. We encourage writing tests for as many scenarios as is possible, even if it means the same lines getting exercised multiple times. It gets us closer to 100% path coverage as well, though we don’t enforce (or measure) that

  • nicoburns  14 hours ago

    > I never claim that 100% coverage has anything to do with code breaking.

    But what I care about is code breaking (or rather, it not breaking). I'd rather put effort into ensuring my test suite does provide a useful benefit in that regard, rather than measure an arbitrary target which is not a good measure of that.

  • xcskier56  17 hours ago

    SimpleCov in ruby has 2 metrics, line coverage and branch coverage. If you really want to be strict, get to 100% branch coverage. This really helps you flesh out all the various scenarios

  • a3w  16 hours ago

    Brakes in cars here in Germany are integrated with less than 50 % coverage in the final model testing that goes to production.

    Seems like even if people could potentially die, industry standards are not really 100% realistic. (Also, redundancy in production is more of a solution than having some failures and recalls, which are solved with money.)