← Back to context

Comment by rcxdude

16 hours ago

Not a good idea for CI tests. It will just make things flaky and gum up your PR/release process. Randomness or any form of nondeterminism should be in a different set of fuzzing tests (if you must use an RNG, a deterministic one is fine for CI).

8 comments

rcxdude

Reply
dathinab  15 hours ago

if it makes thing flaky

then it actually is a huge success

because it found a bug you overlooked in both impl. and tests

at least iff we speak about unit tests

  • jstanley  13 hours ago

    Only if it becomes obvious why it is flaky. If it's just sometimes broken but really hard to reproduce then it just gets piled on to the background level of flakiness and never gets fixed.

    • nomel  12 hours ago

      To get around this, I have it log the relevant inputs, so it can be reproduced.

      The whole concept of allowing a flaky unit test to exist is wild and dangerous to me. It makes a culture of ignoring real failures in what, should be, deterministic code.

      1 reply →

  • jamesfinlayson  3 hours ago

    I remember having a flaky test with random number generation a few years ago - it failed very rarely (like once every few weeks) and when I finally got to fixing it, it was an actual issue (an off by one error).

  • yxhuvud  12 hours ago

    This will often break on stuff like daylight saving changes, while almost as often you don't give a rats ass about the boundary behaviour.