← Back to context

Comment by Maxion

5 days ago

Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?

44 comments

Maxion

Reply
cornholio  4 days ago

The disater plan is to have a few dozens stratum 1 servers spread around the world, each connected to a distinct primary atomic clock, so that a catastrophic disaster needs to take down the global internet itself for all servers to become unreachable.

The failure of a single such server is far from a disaster.

  • sounds  4 days ago

    For those of us near Boulder, it's urgent.

    But the stratum 1 time servers can shrug and route around the damage.

  • Lammy  4 days ago

    And the disaster plan for the disaster plan is to realize that it isn't that important at the human-level to have a clock meticulously set to correspond to other meticulously-set clocks, and that every attempt to force rigid timekeeping on humans is to try to make humans work more like machines rather than to make machines work more like humans.

    • basilgohar  4 days ago

      I really, really can't get behind this sentiment. Having a reliable, accurate time keeping mechanism doesn't seem like an outlandish issue to want to maintain. Timekeeping has been an important mechanism for humans for as long as recorded history. I don't understand the wisdom of shooting down establishing systems to make that better, even if the direct applicability to a single human's life is remote. We are all part of a huge, interconnected system whether we like it or not, and accurate, synchronized timekeeping across the world does not sound nefarious to me.

      3 replies →

    • zettabomb  4 days ago

      Far more things rely on reliable and accurate time-keeping than just being on time to work. Timekeeping is vitally important (even if it's not readily visible) to lots of critical infrastructure worldwide.

      1 reply →

    • Brian_K_White  4 days ago

      This is like the kid in school who doesn't think they should have to learn algebra since they think they will never use it.

    • zzzeek  4 days ago

      oh....no, not really, no, the world needs GPS, so, yeah. this is not like scrooge mcduck telling you to be at work on time. scrooge still has a windup watch

toast0  4 days ago

If access to the site is unsafe and thus the site is closed; not having access seems reasonable.

Time services are available from other locations. That's the disaster plan. I'm sure there will be some negative consequences from this downtime, especially if all the Boulder reference time sources lose power, but disaster plans mitigate negative consequences, they can't eliminate them.

Utility power fails, automatic transfer switches fail, backup generators fail, building fires happen, etc. Sometimes the system has to be shut down.

ssl-3  5 days ago

Maybe this is the disaster plan: There's not a smouldering hole where NIST's Boulder facility used to be, and it will be operational again soon enough.

There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines.

We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland.

This is just one set.

And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.)

And the orbital GPS clocks are controlled by the US Navy, not NIST.

So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth.

Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down.

And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all.

  • Balgair  4 days ago

    There are many backup clocks/clusters that NIST uses as redundancies all around Boulder too, no need to even go up to Fort Collins. As in, NIST has fiber to a few at CU and a few commercial companies, last I checked. They're used in cases just like this one.

    Fun facts about The clock:

    You can't put anything in the room or take anything out. That's how sensitive the clock is.

    The room is just filled with asbestos.

    The actual port for the actual clock, the little metal thingy that is going buzz, buzz, buzz with voltage every second on the dot? Yeah, that little port isn't actually hooked up to anything, as again, it's so sensitive (impedance matching). So they use the other ports on the card for actual data transfer to the rest of the world. They do the adjustments so it's all fine in the end. But you have to define something as the second, and that little unused port is it.

    You can take a few pictures in the cramped little room, but you can't linger, as again, just your extra mass and gravity affects things fairly quickly.

    If there are more questions about time and timekeeping in general, go ahead and ask, though I'll probably get back to them a bit later today.

    • jrronimo  4 days ago

      I'm the Manager of the Computing group at JILA at CU, where utcnist*.colorado.edu used to be housed. Those machines were, for years, consistently the highest bandwidth usage computers on campus.

      Unfortunately, the HP cesium clock that backed the utcnist systems failed a few weeks ago, so they're offline. I believe the plan is to decommission those servers anyway - NIST doesn't even list them on the NTP status page anymore, and Judah Levine has retired (though he still comes in frequently). Judah told me in the past that the typical plan in this situation is that you reference a spare HP clock with the clock at NIST, then drive it over to JILA backed by some sort of battery and put it in the rack, then send in the broken one for refurb (~$20k-$40k; new box is closer to $75k). The same is true for the WWVB station, should its clocks fail.

      There is fiber that connects NIST to CU (it's part of the BRAN - Boulder Research and Administration Network). Typically that's used when comparing some of the new clocks at JILA (like Jun Ye's strontium clock) to NIST's reference. Fun fact: Some years back the group was noticing loss due to the fiber couplers in various closets between JILA & NIST... so they went to the closets and directly spliced the fibers to each other. It's now one single strand of fiber between JILA & NIST Boulder.

      That fiber wasn't connected to the clock that backed utcnist though. utcnist's clock was a commercial cesium clock box from HP that was also fed by GPS. This setup was not particularly sensitive to people being in the room or anything.

      Another fun fact: utcnist3 was an FPGA developed in-house to respond to NTP traffic. Super cool project, though I didn't have anything to do with it, haha.

      4 replies →

    • Workaccount2  4 days ago

      >The actual port for the actual clock, the little metal thingy that is going buzz, buzz, buzz with voltage every second on the dot? Yeah, that little port isn't actually hooked up to anything, as again, it's so sensitive (impedance matching). So they use the other ports on the card for actual data transfer to the rest of the world.

      Can you restate this part in full technical jargon along with more detail? I'm having a hard time following it

      4 replies →

TylerE  5 days ago

Step One of most disaster plans is not to create a second emergency.