But still a kludge. Better: use something equivalent to Go's testing/synctest[0] package, which lets you write tests that run in a bubble where time is fixed and deterministic.
I’ve used freezetime (Python) a decent amount and have experienced some very very very funny flakes due to it.
- Sometimes your test code expects time to be moving forward
- sometimes your code might store classes into a hashmap for caching, and the cache might be built before the freeze time class override kicks in
- sometimes it happens after you have patched the classes and now your cache is weirdly poisoned
- sometimes some serialization code really cares about the exact class used
- sometimes test code acts really weird if time stops moving forward (when people use freezetime frozen=true). Selenium timeouts never clearing was funny
- sometimes your code gets a hold of the unpatched date clsss through silliness but only in one spot
Fun times.
The nicest thing is being able to just pass in a “now” parameter in things that care about time.
- generating test data in a realistic way is often better then hard coding it (also makes it easier to add prop testing or similar)
- make the current time an input to you functions (i.e. the whole old prefer pure functions discussion). This isn't just making things more testable it also can matter to make sure: 1. one unit of logic sees the same time 2. avoid unneeded calls to `now()` (only rarely matters, but can matter)
Similarly, I like .NET's TimeProvider abstraction [1]. You pass a TimeProvider to your functions. At runtime you can provide the default TimeProvider.System. When testing FakeTimeProvider has a lot of handy tools to do deterministic testing.
One of the further benefits of .NET's TimeProvider is that it can also be provided to low level async methods like `await Task.Delay(time, timeProvider, cancellationToken)` which also increases the testability of general asynchronous code in a deterministic sandbox once you learn to pass TimeProvider to even low level calls that take an optional one.
I always wanted to make feature flags system where each FF must declare an expiration date max 1 year in the future and start failing CI beyond that date to force someone to reevaluate and clean up.
It's just too easy to keep adding new feature flags and never removing them. Until one day the FF backend goes down and you have 300 FFs all evaluate to false.
We had something like this where I last worked. Whenever we were adding new features or adding things that had potential for significant regressions, we were expected to add feature flags around the change/addition and set an expiration date for three months or so in advance. Once that rolled around, we'd either remove the old path or evaluate if it was necessary to have around as a permanent feature.
I think it worked out really well even though it increased the administrative overhead. We were always able to quickly revert behavior without needing to push code and it let us gradually shrink a lot of the legacy features we had on the project.
We've done that at a few places I've been at - it's tricky because if the failure is too short its just annoying toil, but if it's too long there's risk of losing context and having to remember what the heck we were thinking.
Overall it's still net positive for me in certain cases of enforcing things to be temporary, or at least revisited.
I mean, sure, that can happen, but that obviously depends on what the test is testing, it's not like it's bad in all cases to say "now plus 1 year". In the case in question it's really just "cookie is far enough in the future so it hasn't expired", so "expire X years in the future from now" is fine.
Arguably you should have a fixed start date for any given test, but time is quite hard to abstract out like that (there's enough time APIs you'd want OS support, but linux for example doesn't support clock namespaces for the realtime clock, only a few monotonic clocks)
Not a good idea for CI tests. It will just make things flaky and gum up your PR/release process. Randomness or any form of nondeterminism should be in a different set of fuzzing tests (if you must use an RNG, a deterministic one is fine for CI).
If this isn't a joke, I'd be very interested in the reasoning behind that statement, and whether or not there are some qualifications on when it applies.
Generate fuzz tests using random values with a fixed seed, sure, but using random values in tests that run on CI seems like a recipe for hard-to-reproduce flaky builds unless you have really good logging.
Are you joking? This is the kind of thing that leads to flaky tests. I was always counseled against the use of randomness in my tests, unless we're talking generative testing like quickcheck.
I've got some tests in active code bases that are using the end of 32-bit Unix time as "we'll never get there". That's not because the devs were lazy, these tests date from when that was the best they could possibly do. They're on track to be cycled out well before then (hopefully this year), so, hopefully, they'll be right that their code "won't get there"... but then there's the testing and code that assumes this that I don't know about that may still be a problem.
"End of Unix time" is under 12 years now, so, a bit longer than the time frame of this test, but we're coming up on it.
Now I feel bad for using (system foundation timestamp)+100 years as end of "forever" ownership relations in one of my systems. Looking now, it's only 89 years left. I think I should use nulls instead.
Most updates to avoid the 2038 problem really just delay it until 10889. Maybe in eight in a half millennia, they will have figured out something that lasts longer.
As others have stated, the lack of visible effect is not the same thing as there never having been a land mine in the first place.
I can tell you anecdotally that on 12/31/2000 I was hanging with some friends. At 12PM UTC we turned on the footage from London. At first it appeared to be a fiery hellscape armageddon. while it turned out to just be fireworks with a wierd camera angle, there was a moment where we were concerned something was actually happening. Most of us in the room were technologists, and while we figured it'd all be no big deal, we weren't *sure* and it very much alarmed us to see it on the screen.
> At the Great Midnight at the century's end, signifying culture will flip over into a number-based counterculture, retroprocessing the last 100 years. Whether global disaster ensues or not, Y2K is a singularity for cybernetic culture. It's time to get Y2K positive.
While there was a lot of FUD in the media, there were also a lot of scenarios that were actually possible but were averted due to a LOT of work and attention ahead of time. It should be looked at, IMO, as a success of communication, warnings, and a lot of effort that nothing of major significance happened.
i had to plant a 10 year time bomb in our SAML SP certificate because AFAIK there is no other way to do it. It’s been 7 years since then. Dreading contacting all the IDPs and getting them to update the SAML config.
But before you judge the fix too hashly, I bet it’s just a quick and easy fix that will suffice while a proper fix (to avoid depending on external state) is written.
of course it is just an easy fix. it's the kind of solution that even someone like me could write who has no understanding of the code a all. (i am not trying to imply that the submitter of the PR doesn't understand the code, just that understanding it is unlikely to be necessary, thus the change bears no risk.
but, the solution now hides the problem. if i wanted to get someone to solve the problem i'd set the new date in the near future until someone gets annoyed enough to fix it for real.
and i have to ask, why is this a hardcoded date at all? why not "now plus one week"?
There’s a lot to be said for simplicity. The more logic you put into handling the dates correctly in the tests, the more likely you are to mess up the tests themselves. These tests were easy to write, easy to review, easy to verify, and served perfectly well for 10 years.
Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
> Us, ten years after generating the certificate: "Who could have possibly foreseen that a computer science department would still be here ten years later."
This was why there was a Y2K bug. Most of that code was written in the 80s, during the Reagan era. Nobody expected civilization to make it to the year 2000.
No, people thought that storing a year as two digits was fine because computers were advancing so fast that it was unlikely they'd still be used in the year 2000 - or if they were it was someone else's problem.
And they were mostly right! Not many 80s machines were still being used in 1999, but lots of software that had roots to then was being used. Data formats and such have a tendency to stick around.
Software has incredible inertia compared to hardware.
It is effectively trivial to buy millions of dollars of hardware to upgrade your stuff when compared with paying for existing software to be rewritten for a new platform.
But still a kludge. Better: use something equivalent to Go's testing/synctest[0] package, which lets you write tests that run in a bubble where time is fixed and deterministic.
[0] https://pkg.go.dev/testing/synctest
I’ve used freezetime (Python) a decent amount and have experienced some very very very funny flakes due to it.
- Sometimes your test code expects time to be moving forward
- sometimes your code might store classes into a hashmap for caching, and the cache might be built before the freeze time class override kicks in
- sometimes it happens after you have patched the classes and now your cache is weirdly poisoned
- sometimes some serialization code really cares about the exact class used
- sometimes test code acts really weird if time stops moving forward (when people use freezetime frozen=true). Selenium timeouts never clearing was funny
- sometimes your code gets a hold of the unpatched date clsss through silliness but only in one spot
Fun times.
The nicest thing is being able to just pass in a “now” parameter in things that care about time.
in general
- generating test data in a realistic way is often better then hard coding it (also makes it easier to add prop testing or similar)
- make the current time an input to you functions (i.e. the whole old prefer pure functions discussion). This isn't just making things more testable it also can matter to make sure: 1. one unit of logic sees the same time 2. avoid unneeded calls to `now()` (only rarely matters, but can matter)
Similarly, I like .NET's TimeProvider abstraction [1]. You pass a TimeProvider to your functions. At runtime you can provide the default TimeProvider.System. When testing FakeTimeProvider has a lot of handy tools to do deterministic testing.
One of the further benefits of .NET's TimeProvider is that it can also be provided to low level async methods like `await Task.Delay(time, timeProvider, cancellationToken)` which also increases the testability of general asynchronous code in a deterministic sandbox once you learn to pass TimeProvider to even low level calls that take an optional one.
[1] https://learn.microsoft.com/en-us/dotnet/standard/datetime/t...
2 replies →
Also, if you do use `now()` in this case you can always do `now() + SomeDistantDuration`
Ruby's timecop is great for those test scenarios https://github.com/travisjeffery/timecop
Doesn't that just turn bugs in test in n years into bugs in prod in n years?
That seems like a downgrade to me!
No, because prod doesn't have hardcoded cookies baked into it?
2 replies →
libfaketime is cool for testing this kind of thing too.
Not as convenient for unit tests cause you have to run the test with LD_PRELOAD.
Found this one for rust: https://github.com/museun/mock_instant
This can cause other types of bugs to go unnoticed, such as leap year fun (if you handle 100 years, did you handle the 400th year?).
[dead]
Interesting, from the title I thought it was intentional, as a "forced code review." Apparently not, but now I really like that idea!
I always wanted to make feature flags system where each FF must declare an expiration date max 1 year in the future and start failing CI beyond that date to force someone to reevaluate and clean up.
It's just too easy to keep adding new feature flags and never removing them. Until one day the FF backend goes down and you have 300 FFs all evaluate to false.
We had something like this where I last worked. Whenever we were adding new features or adding things that had potential for significant regressions, we were expected to add feature flags around the change/addition and set an expiration date for three months or so in advance. Once that rolled around, we'd either remove the old path or evaluate if it was necessary to have around as a permanent feature.
I think it worked out really well even though it increased the administrative overhead. We were always able to quickly revert behavior without needing to push code and it let us gradually shrink a lot of the legacy features we had on the project.
We've done that at a few places I've been at - it's tricky because if the failure is too short its just annoying toil, but if it's too long there's risk of losing context and having to remember what the heck we were thinking.
Overall it's still net positive for me in certain cases of enforcing things to be temporary, or at least revisited.
Which is why SSL certs are now 47 days long or whatever it is.
1 reply →
Just skimmed the PR, I'm sure the author knows more than I - but why hard code a date at all? Why not do something like `today + 1 year`?
That can easily lead to breaking tests due to time-zones, daylight saving time or the variable length of months.
We experienced several of those over the years, and generally it was the test that was wrong, not the code it was testing.
For example, this simplified test hits several of those pitfalls:
I mean, sure, that can happen, but that obviously depends on what the test is testing, it's not like it's bad in all cases to say "now plus 1 year". In the case in question it's really just "cookie is far enough in the future so it hasn't expired", so "expire X years in the future from now" is fine.
That introduces dependency of a clock which might be undesirable, just had a similar problem where i also went for hardcoding for that reason.
There's already a clock dependency. The test fails because of that.
Arguably you should have a fixed start date for any given test, but time is quite hard to abstract out like that (there's enough time APIs you'd want OS support, but linux for example doesn't support clock namespaces for the realtime clock, only a few monotonic clocks)
Because it should be `today + 1 year + randomInt(1,42) days`.
Always include some randomness in test values.
Not a good idea for CI tests. It will just make things flaky and gum up your PR/release process. Randomness or any form of nondeterminism should be in a different set of fuzzing tests (if you must use an RNG, a deterministic one is fine for CI).
8 replies →
> Always include some randomness in test values.
If this isn't a joke, I'd be very interested in the reasoning behind that statement, and whether or not there are some qualifications on when it applies.
14 replies →
Generate fuzz tests using random values with a fixed seed, sure, but using random values in tests that run on CI seems like a recipe for hard-to-reproduce flaky builds unless you have really good logging.
Interesting, haven't heard this before (I don't know much about testing). Is this kind of like fuzzing?
1 reply →
Are you joking? This is the kind of thing that leads to flaky tests. I was always counseled against the use of randomness in my tests, unless we're talking generative testing like quickcheck.
6 replies →
Any time constant will be exceeded someday.
An impossibly short period of time after the heat death of the universe on a system that shouldn’t even exist: ERROR TIME_TEST FAILURE
Posted on HN in 2126: 100 years ago, someone wrote a test for servo that included an expiry in 2126
I've got some tests in active code bases that are using the end of 32-bit Unix time as "we'll never get there". That's not because the devs were lazy, these tests date from when that was the best they could possibly do. They're on track to be cycled out well before then (hopefully this year), so, hopefully, they'll be right that their code "won't get there"... but then there's the testing and code that assumes this that I don't know about that may still be a problem.
"End of Unix time" is under 12 years now, so, a bit longer than the time frame of this test, but we're coming up on it.
1 reply →
Now I feel bad for using (system foundation timestamp)+100 years as end of "forever" ownership relations in one of my systems. Looking now, it's only 89 years left. I think I should use nulls instead.
3 replies →
This is why I always use the year 2525. Not my problem, assuming man is still alive.
Most updates to avoid the 2038 problem really just delay it until 10889. Maybe in eight in a half millennia, they will have figured out something that lasts longer.
How is 10889 a problem? I thought the move to 64 bit added billions of years.
1 reply →
Yep - that's why I always choose my time constants to be during years when I will be retired, or possibly dead.
If you're going to kick the can down the road, why not kick it pretty far?
Who here remembers the fud of Y2K?
Don't mistake a defused bomb for a dud.
https://en.wikipedia.org/wiki/Preparedness_paradox
1 reply →
I remember the reality of all the work needed to avoid issues.
As others have stated, the lack of visible effect is not the same thing as there never having been a land mine in the first place.
I can tell you anecdotally that on 12/31/2000 I was hanging with some friends. At 12PM UTC we turned on the footage from London. At first it appeared to be a fiery hellscape armageddon. while it turned out to just be fireworks with a wierd camera angle, there was a moment where we were concerned something was actually happening. Most of us in the room were technologists, and while we figured it'd all be no big deal, we weren't *sure* and it very much alarmed us to see it on the screen.
Made me think of Mark Fisher's Y2K Positive text:
> At the Great Midnight at the century's end, signifying culture will flip over into a number-based counterculture, retroprocessing the last 100 years. Whether global disaster ensues or not, Y2K is a singularity for cybernetic culture. It's time to get Y2K positive.
Mark Fisher (2004). Y2K Positive in Mute.
While there was a lot of FUD in the media, there were also a lot of scenarios that were actually possible but were averted due to a LOT of work and attention ahead of time. It should be looked at, IMO, as a success of communication, warnings, and a lot of effort that nothing of major significance happened.
8 replies →
Tell us you weren't involved in Y2K iwithout telling us you weren't involved in Y2K.
Exciting times with an anticlimactic end; I was in middle school, relishing the chaos of the adult world.
Another victim of the preparedness paradox.
i had to plant a 10 year time bomb in our SAML SP certificate because AFAIK there is no other way to do it. It’s been 7 years since then. Dreading contacting all the IDPs and getting them to update the SAML config.
Classic!
But before you judge the fix too hashly, I bet it’s just a quick and easy fix that will suffice while a proper fix (to avoid depending on external state) is written.
I'll bet you one US Dollar that this is a scenario where the temporary fix becomes the permanent one. (Well, at least, permanent for a hundred years.)
Some day, Pham Nuwen is going to be bitching about this test suite between a pair of star systems.
That’s one of my favorite books :)
I agree that it’s plausible!
of course it is just an easy fix. it's the kind of solution that even someone like me could write who has no understanding of the code a all. (i am not trying to imply that the submitter of the PR doesn't understand the code, just that understanding it is unlikely to be necessary, thus the change bears no risk.
but, the solution now hides the problem. if i wanted to get someone to solve the problem i'd set the new date in the near future until someone gets annoyed enough to fix it for real.
and i have to ask, why is this a hardcoded date at all? why not "now plus one week"?
There’s a lot to be said for simplicity. The more logic you put into handling the dates correctly in the tests, the more likely you are to mess up the tests themselves. These tests were easy to write, easy to review, easy to verify, and served perfectly well for 10 years.
But doing it right shouldn’t be all that hard.
A comment from the PR
> Not a serious problem, but the weekdays are wrong. For example, 18-Apr-2127 is a Friday, not Sunday.
There is now many magical dates to remember - 2126 ( I think PR was updated after that comment) and 2177. There is also 2028 also somewhere.
I fixed one of these test cases too. Attached to it was a comment:
Alas, he was still working, albeit at another firm.
“Someone” please stop write Someone at every possible post, especially on X.
quite the nostalgic test to fix lol
what a nostalgic test to fix lol
[flagged]
Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
https://news.ycombinator.com/newsguidelines.html
It was started by people who thought Twitter didn't have enough censorship (back when it had a lot more).
I guess that's a matter of personal sensibilities, but it's pretty funny to me.
(Note: this is the only fact I know about it, happy to learn more.)
Any social space will break down upon reaching a critical point in representation of the general populace.
I have no idea about the development however.
Worked for me.
One of the comments:
> Us, ten years after generating the certificate: "Who could have possibly foreseen that a computer science department would still be here ten years later."
This was why there was a Y2K bug. Most of that code was written in the 80s, during the Reagan era. Nobody expected civilization to make it to the year 2000.
No, people thought that storing a year as two digits was fine because computers were advancing so fast that it was unlikely they'd still be used in the year 2000 - or if they were it was someone else's problem.
And they were mostly right! Not many 80s machines were still being used in 1999, but lots of software that had roots to then was being used. Data formats and such have a tendency to stick around.
Software has incredible inertia compared to hardware.
It is effectively trivial to buy millions of dollars of hardware to upgrade your stuff when compared with paying for existing software to be rewritten for a new platform.
2 replies →