Comment by Alupis

21 hours ago

Just skimmed the PR, I'm sure the author knows more than I - but why hard code a date at all? Why not do something like `today + 1 year`?

40 comments

Alupis

Reply
CodesInChaos  16 hours ago

That can easily lead to breaking tests due to time-zones, daylight saving time or the variable length of months.

We experienced several of those over the years, and generally it was the test that was wrong, not the code it was testing.

For example, this simplified test hits several of those pitfalls:

    var expected = start.AddMonths(1);
    var actual = start.ToLocal().AddMonths(1).ToUtc();
    Assert(expected == actual);

  • hn_throwaway_99  7 hours ago

    I mean, sure, that can happen, but that obviously depends on what the test is testing, it's not like it's bad in all cases to say "now plus 1 year". In the case in question it's really just "cookie is far enough in the future so it hasn't expired", so "expire X years in the future from now" is fine.

johanvts  20 hours ago

That introduces dependency of a clock which might be undesirable, just had a similar problem where i also went for hardcoding for that reason.

  • rcxdude  20 hours ago

    Arguably you should have a fixed start date for any given test, but time is quite hard to abstract out like that (there's enough time APIs you'd want OS support, but linux for example doesn't support clock namespaces for the realtime clock, only a few monotonic clocks)

whynotmaybe  20 hours ago

Because it should be `today + 1 year + randomInt(1,42) days`.

Always include some randomness in test values.

  • rcxdude  20 hours ago

    Not a good idea for CI tests. It will just make things flaky and gum up your PR/release process. Randomness or any form of nondeterminism should be in a different set of fuzzing tests (if you must use an RNG, a deterministic one is fine for CI).

  • zelos  19 hours ago

    Generate fuzz tests using random values with a fixed seed, sure, but using random values in tests that run on CI seems like a recipe for hard-to-reproduce flaky builds unless you have really good logging.

  • CoastalCoder  20 hours ago

    > Always include some randomness in test values.

    If this isn't a joke, I'd be very interested in the reasoning behind that statement, and whether or not there are some qualifications on when it applies.

    • j1mr10rd4n  11 hours ago

      There's another good reason that hasn't been detailed in the comments so far: expressing intent.

      A test should communicate its reason for testing the subject, and when an input is generated or random, it clearly communicates that this test doesn't care about the specific _value_ of that input, it's focussed on something else.

      This has other beneficial effects on test suites, especially as they change over the lifetime of their subjects:

      * keeping test data isolated, avoiding coupling across tests * avoiding magic strings * and as mentioned in this thread, any "flakiness" is probably a signal of an edge-case that should be handled deterministically and * it's more fun [1]

      [1] https://arxiv.org/pdf/2312.01680

    • dathinab  19 hours ago

      humans are very good at overlooking edge cases, off by one errors etc.

      so if you generate test data randomly you have a higher chance of "accidentally" running into overlooked edge cases

      you could say there is a "adding more random -> cost" ladder like

      - no randomness, no cost, nothing gained

      - a bit of randomness, very small cost, very rarely beneficial (<- doable in unit tests)

      - (limited) prop testing, high cost (test runs multiple times with many random values), decent chance to find incorrect edge cases (<- can be barely doable in unit tests, if limited enough, often feature gates as too expensive)

      - (full) prop testing/fuzzing, very very high cost, very high chance incorrect edge cases are found IFF the domain isn't too large (<- a full test run might need days to complete)

      6 replies →

    • whynotmaybe  20 hours ago

      Must be some Mandela effect about some TDD documentation I read a long time ago.

      If you test math_add(1,2) and it returns 3, you don't know if the code does `return 3` or `return x+y`.

      It seems I might need to revise my view.

      5 replies →

  • andai  20 hours ago

    Interesting, haven't heard this before (I don't know much about testing). Is this kind of like fuzzing?

    • whynotmaybe  20 hours ago

      I recently had race condition that made tests randomly fail because one test created "data_1" and another test also created "data_1".

      - Test 1 -> set data_1 with value 1

      - Test 1 -> `do some magic`

      - Test 1 -> assert value 1 + magic = expected value

      - Test 2 -> set data_1 with value 2

      But this can fail if `do some magic` is slow and Test 2 starts before Test 1 asserts.

      So I can either stop parallelism, but in real life parallelism exists, or ensure that each test as random id, just like it would happen in real life.

  • devin  20 hours ago

    Are you joking? This is the kind of thing that leads to flaky tests. I was always counseled against the use of randomness in my tests, unless we're talking generative testing like quickcheck.