← Back to context

Comment by eithed

8 hours ago

These were my thoughts as well and it's nothing new, I think, regarding testing altogether:

- testing libraries (and in this case - language itself) can have bugs

- what is not covered by tests can have bugs

Additionally would add that tests verify the assumptions of coder, not expectations of the business.

To give benefit to the author - I'd read the article as: having tests for given thing ensures that it does the thing that you built the tests for. This doesn't mean that your application is free of bugs (unless you have 100% coverage, can control entire state of the system, etc) nor that it does the right thing (or that it does the thing the right way)

3 comments

eithed

Reply
necovek  4 hours ago

I like to differentiate between coverage by lines and semantic coverage: sometimes you need to exercise a single line multiple times to get full semantic coverage, and better semantic coverage usually beats larger line coverage for detecting problems and preventing regressions.

  • eithed  4 hours ago

    I think mutation testing helps in defining semantic coverage, if I understand what you're saying

    • necovek  1 hour ago

      Yes, mutation testing and similar techniques like fuzzing can help with it, but sometimes you want to be more deterministic: there are usually a lot of hidden side effects that are not obvious, and probably a source of majority of software bugs today.

      Eg. something as simple as

        function foo(url) {
     data = fetch(url);
     return data;
  }

      has a bunch of exceptions that can happen in fetch() not covered with your tests, yet you can get 100% line coverage with a single deterministic happy path test.

      Basically, any non-functional side-effect behavior is a source of semantic "hiding" when measuring test coverage by lines being executed, and there is usually a lot of it (logging is another common source of side-effects not tested well). Some languages can handle this better with their typing approaches, but ultimately, there will be things that can behave differently depending on external circumstances (even bit flips, OOMs, full disk...) that were not planned for and do not flow from the code itself.