Comment by 7402
7 years ago
This was the part that absolutely stunned me:
> Last fall, the night before daylight-saving time ended, an all-user e-mail alert went out. The system did not have a way to record information when the hour from 1 a.m. to 1:59 a.m. repeated in the night. This was, for the system, a surprise event. The only solution was to shut down the lab systems during the repeated hour.
Any system like this that fails to store all times as UTC is broken.
No one should make this mistake more than once. I made it once, and boy was it painful. And boy did I feel stupid to realize I had forgotten about daylight savings time. I just can't see how it was possible for the designers of this huge system to miss this.
At least a few years ago, when I was at Epic (not on a medical-user-facing team), the issue that caused this wasn't the data storage of the timestamps or whatever. It's figuring out how to display events during that repeated hour in charts that assume 24 hour days in patient-safe way.
Say you see on the chart that a patient received medication at 1:30. It's now 2:30, and the patient is supposed to receive medication every 2 hours. Do you give them another dose? Did they get their does at the first 1:30 or the 2nd 1:30? Expand that by the literal thousands of super (almost frustratingly) customizable workflows, custom charts, etc., and the problem gets pretty big.
Considering most (maybe all?) Epic upgrades require brief downtimes, a lot of hospitals chose to take their downtime to install updates during the DST changeover time and rely on their doctors/nurses/etc. to be extra careful on paper during the doubled hour. Is it a great system? Not really. But it kind of works I guess.
It's not always necessary or desirable to store all times as UTC. What you actually need is a time plus offset from UTC. Storing times as UTC is essentially just a special case of that where the offset is always 0.
Most lab data is communicated to other systems using HL7 V2.x messages. The standard does allow for including a time zone offset in all date/time fields. However, many older implementations fail to include the offset and just use local time. Obviously that's not a good idea.
>It's not always necessary or desirable to store all times as UTC. What you actually need is a time plus offset from UTC. Storing times as UTC is essentially just a special case of that where the offset is always 0.
It depends on the needs of the system. For example, a patient comes in, and has a test or takes a medication at 8am local time (12 noon UTC). Now the patient moves one timezone east, and their new Dr. looks at that test/medication record. How do you display the time of that test? 8am (when the test was taken local time), or 9am (the wall clock time in the new timezone). If you care about waiting 24 hours before the next test, the latter is what you want. If you want to know how early in the patient's morning the test was taken, you prefer the former.
You are making a bad assumption here: "Any system" ... Chances are we are talking about multiple systems created at different times and by different companies.
I could easily see a case where the timestamp is stored as a string to interoperate across various computer SYSTEMS.
Doing a wide scale fix for this problem is probably cost-prohibitive - assuming it is even possible.
From experience with an EHR system: these things are so big that interoperating between different pieces of big-ball-of-mud systems is a dicey matter and making any changes requires extensive manual tests, since there are approximately zero automated tests.
I have a feeling that lots of stuff that translates between a human's conception of time and a computer timestamp have problems with DST. Have you ever tried editing a crontab entry in the hour leading up to the DST change? It won't run until after 2am (at least on RHEL7). I found this out last weekend during the time change.
I contracted at Cerner, which is Epic's largest competitor, for a couple years. I spent a solid three months on the Android implementation of a time function. The time you mention is called the "ambiguous hour" and everyone working on similar problems on my team was well aware of the issue and all unit testing scenarios accounted for the issue. I can't imagine that Epic completely ignored or was unaware of this.
From experience, at least one EHR system doesn't use UTC because it has never used UTC and is an amalgamation of code going back to the 80s. This can probably never be fixed because reliance on this format is threaded throughout the entire system, which has approximately no unit tests.
It's broken, but it's been broken in this way for decades and hasn't stopped it being very successfully sold.
Is this the one that can’t handle installations across time zones?