Comment by boulos

24 days ago

This is nice work, but I found the bug finding example to be weird:

> One such bug was in the sign function for zigzag decoding of the datrs/varinteger library. On input Std.U64.MAX, the expression (value + 1) overflowed, causing crashes in debug mode and silent corruption in release mode—an edge case that testing and fuzzing would typically miss.

In what way would this boundary condition case be considered something that "testing [...] would typically miss"? It's certainly something that bad tests would miss or not think about, but I find that (a) careful people and (b) ML coding systems are actually really good at "oh, I should test the extreme values". Especially for things that parse user input.

I'm curious if they found other bugs that were more interesting, but found them too hard to explain quickly.

9 comments

boulos

Reply
Groxx  24 days ago

particularly "and fuzzing", yea. fuzzing generally does intentionally explore boundary values, from what I've seen. for an encoding library like this, I think it's fair to say that fuzzing is a baseline expectation for any decent code, and it almost certainly would've caught this in seconds.

--- edit

concretely, I made a very simple round-trip test with proptest, and got dozens of failures and this in less than a second:

    thread 'signed_round_trip' (50528) panicked at tests/test.rs:72:1:
    Test failed: attempt to multiply with overflow.
    minimal failing input: value = 4611686018427387904
        successes: 2
        local rejects: 0
        global rejects: 0

fjdjshsh  24 days ago

Maybe it's not something they would "typically miss", but, from proof by existence, it's something they sometimes miss.

It does speak to the benefits of using lean in that you don't need to be clever about the different examples you test.

Quekid5  24 days ago

Because this is garbage PR. That's it.

Every property-based testing system (invented ca. 1980) will explore boundary values. The semantics (or lack thereof) of C and C++ can make this difficult to actually test for because the compiler is allowed to say "test passed" to any input leading to UB.

  • vlovich123  24 days ago

    Property based testing is good at generating boundary values for inputs. But for any more complicated piece of code getting boundary value coverage of interior values is an open problem that requires instrumentation feedback to understand branch coverage and value coverage of the code that got tested. It’s not an easy thing at all.

  • teiferer  23 days ago

    > The semantics (or lack thereof) of C and C++ can make this difficult to actually test for because the compiler is allowed to say "test passed" to any input leading to UB.

    I get what you are saying but does this actually apply to a test? If the code under test is in one compilation unit and the test harness in another and they are linked together then the UB optimization issue ends at the API boundary and can't possibly make the test pass ..?

    • Quekid5  12 days ago

      It's definitely less likely with that technique (in practice!), but still very possible, via e.g. ODR violation or even the classic SIOF.

      So, ideally you'd want each separate test case to be compiled separately, but even then you wouldn't be safe! ... because any UB in that test (or the code it's testing!) could lead to a random pass.

      UB is good in some ways, but other ways it's really really bad.

      EDIT: I will say: If you have a UBSAN turned on for testing, etc. you're reasonably safe... but not fully. There's a lot of stuff they don't catch because it's essentially impossible.

Exoristos  24 days ago

Yes, it's basic QA. If tests missed this kind of thing, they would be of much more limited use than we generally expect them to be. It raises questions about the authors' background.