Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.
Notably, we had the marshal fire here 4 years ago and recently Xcel settled for $680M for their role in the fire. So they're probably pretty keen not to be on the hook again
I guess that explains why they had no qualms shutting down half of Boulder's power with a vague time horizon. After losing everything in my fridge, though, they finally turned it back on today.
tl;dr - the fire destroyed over 1,000 homes, two deaths. The local electrical utility, Xcel, was found as a contributing cause from sparking power lines during a strong wind storm. As a result, electrical utilities now cut power to affected areas during strong winds.
The disater plan is to have a few dozens stratum 1 servers spread around the world, each connected to a distinct primary atomic clock, so that a catastrophic disaster needs to take down the global internet itself for all servers to become unreachable.
The failure of a single such server is far from a disaster.
If access to the site is unsafe and thus the site is closed; not having access seems reasonable.
Time services are available from other locations. That's the disaster plan. I'm sure there will be some negative consequences from this downtime, especially if all the Boulder reference time sources lose power, but disaster plans mitigate negative consequences, they can't eliminate them.
Utility power fails, automatic transfer switches fail, backup generators fail, building fires happen, etc. Sometimes the system has to be shut down.
Maybe this is the disaster plan: There's not a smouldering hole where NIST's Boulder facility used to be, and it will be operational again soon enough.
There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines.
We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland.
This is just one set.
And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.)
And the orbital GPS clocks are controlled by the US Navy, not NIST.
So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth.
Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down.
And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all.
Of the various internet .+P, NTP is one I never learned about as a student, so now I'm looking at its web page [1] by its creator David L. Mills (1938-2024). I've found one video of him giving a retrospective of his extensive internet work; he talks about NTP at 34:51 [2] and later at 56:26 [3].
In [3] he mentions that one can use NTP to observe frequency deviations and use it as an early warning system for fire and AC failure. That really intrigues me. Can you actually? Has this ever been implemented?
Oscillators of all kinds are temperature dependent.
That's why the most stable ones are insulated and ovenized[1].
So an AC failure which would lead to higher room temperatures would lead to stronger or more frequent correction by the NTP client, as the local oscillator would drift more.
Not sure about the fire case though. I mean the same applies there but I'm not imaginative enough to think of a realistic scenario where NTP would be useful for averting a fire.
I knew of some experiments in this space back in the late 1980s or early 1990s - but it was specifically with DECstation hardware that had terrible clocks (not used for alerting, just "this graphs nicely against temperature".) https://groups.csail.mit.edu/ana/Publications/PubPDFs/Greg.T... (PDF) 4.2.1 does talk about explaining local clock frequency changes with office temperature changes (because they overwhelm a clock-aging model) but it doesn't have graphs so perhaps they weren't clear enough to include (or just not relevant enough to Time Surveying.)
> Facility operators anticipated needing to shutdown the heat-exchange infrastructure providing air cooling to many parts of the building, including some internal networking closets. As a result, many of these too were preemptively shutdown with the result that our group lacks much of the monitoring and control capabilities we ordinarily have
Having a parallel low bandwidth, low power, low waste heat network infrastructure for this suddenly seems useful.
NIST campus status: Due to elevated fire risk and a power outage for the Boulder area, the DOC Boulder Labs campus is CLOSED on December 19 for onsite business and no public access is permitted; previously approved accesses are revoked.[1]
WWV still seems to be up, including voice phone access.
NIST Boulder has a recorded phone number for site status, and it says that as of December 20, the site is closed with no access.
NIST's main web site says they put status info on various social media accounts, but there's no announcement about this.
Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.
Same for database transaction roll back and roll forward actions.
And most enterprises, including banks, use databases.
So by bad luck, you may get a couple of transactions reversed in order of time, such as a $20 debit incorrectly happening before a $10 credit, when your bank balance was only $10 prior to both those transactions. So your balance temporarily goes negative.
Now imagine if all those amounts were ten thousand times higher ...
Google has their own fleet of atomic clocks and time servers. So does AWS. So does Microsoft. So does Ubuntu. They're not going to drift enough for months to cause trouble. So the Internet can ride through this, mostly.
The main problem will be services that assume at least one of the NIST time servers is up. Somewhere, there's going to be something that won't work right when all the NIST NTP servers are down. But what?
Ubuntu using atomic clocks would surprise me. Sure they could, but it's not obvious to me why they would spend $$$$ on such. More plausible to me seems that they would be using GPSDO as reference clocks (in this context, about as good as your own atomic clock), iff they were running their own time servers. Google finds only that they are using servers from the NTP Pool Project, which will be using a variety of reference clocks.
If you have information on what they actually are using internally, please share.
NIST maintains several time standards. Gaithersburg MD is still up and I assume Hawaii is as well. Other than potential damage to equipment from loss of power (turbo molecular vacuum pumps and oil diffusion pumps might end up failing in interesting ways if not shut down properly) it will just take some time for the clocks to be recalibrated against the other NIST standards.
No noteworthy impact at all. The NTP network has hundreds to thousands of redundant servers and hundreds of redundant reference clocks.
The network will route around the damage with no real effects. Maybe a few microseconds of jitter as you have to ask a more distant server for the time.
FYI, this was posted a month ago when discussing thermal effects of clock drift. I thought it was quite interesting view of what the WWVB location looks like:
This makes me wonder, if you take the average time of all wristwatches on the planet, accounting for timezones and throwing out outliers, how close would you get to NTP time?
And how many randomly chosen wristwatches would you need to get anything reasonable?
I have a hunch my casio wrist watch is designed to be running a bit too quick to make resetting the seconds easier.
Your averaging assumes manufacturers try to make their watches as accurate as possible for average conditions
I think this comment is referencing the government's recent announcement[0] to shut down the National Center for Atmospheric Research in Boulder. They do climate research at the Mesa Laboratory there.
It's open to the public for visits. They have a small science museum, offices, a library, etc. I highly recommend anyone with interest and opportunity to visit the mesa lab soon. It may not be open much longer. The view alone is worth the trip, and the building is cool too.
We had some fun requesting key for accessing nist time servers. the process is (quoted from website)
NIST currently offers this service free of charge. We require written requests to arrive by U.S. mail or fax containing:
Your organization’s name, physical address, fax number (if desired as a reply method).
One or more point-of-contact personnel or system operators authorized to receive key data and other correspondence: names, phone numbers, email addresses.
Up to four static IPv4 network addresses under the user’s control which will be allowed to use the unique key. By special arrangement, additional addresses or address ranges may be requested.
Desired hash function (“key type”). NIST currently supports MD5, SHA1, SHA256, and HMAC-SHA256. Please list any limitations your client software places on key values, if known: maximum length, characters used, or whether hexadecimal key representations are required. If you prefer, please share details about your client software or NTP appliance so we can anticipate key format issues.
Desired method for NIST’s reply: U.S. mail, fax, or a secure download service operated by Department of Commerce.
NIST will not use email for sending key data.
ps. there actually seems to be improvement over what they had year ago. they added "secure download service". and previously they had message that nobody assigned to actively monitor mailbox so if you didn't get key, please email us so we will check it
Can anybody speak to the current best practices around running underground power lines? I see these types of articles about above-ground distribution systems from time-to-time, particularly in California. I feel lucky that my area has underground power, but that was installed back in the 1980s. Would it be prohibitively expensive for Boulder’s utility provider to move to underground distribution? I can’t help but think it could be worth the cost to reduce wildfire risk and offer more reliable service.
Think of it like this: overhead power lines require you to dig a 5-7’ deep hole that’s 2’ in diameter every 90’.
Underground power supplied through cable requires you to bury the cable minimum 3’ in the ground in rigid ductwork the entire 90’. Any time that cable runs under a roadway that ductwork needs to be encased in concrete. In urban and semi urban areas you also compete with other buried infrastructure for space - sewer, city/municipal infrastructure, gas, electrical transmission, etc.
While underground distribution systems are less prone to interruption from bad weather it depends on the circuit design. If the underground portion of the circuit is fed from overhead power lines coming from the distribution substation you will still experience interruptions from faults on the overhead. These faults can also occur on overhead transmission circuits (the lines feeding the distribution substations and/or very large industrial customers).
Underground distribution comes at a cost premium compared to overhead distribution. It’s akin to the cost of building a picket fence vs installing a geothermal heating system for your home. This is why new sub divisions will commonly have underground cable installed as the entire neighborhood is being constructed - there’s no need to retrofit underground cable into an existing area and so the costs are lower and borne upfront.
It’s more cost effective for them to turn the power off as a storm rolls through, patrol, make repairs and reenergize then to move everything underground. Lost revenue during that period is a small fraction of the cost of taking an existing grid and rebuilding it underground. This is especially true for transmission circuits that are strung between steel towers over enormous distances.
Germany here, never heard of any issues regarding underground power (or phone) lines. Ultra High voltage (distribution network) is above ground here, but no issued with that either.
There's two other sites for the time.nist.gov service so it'll be okay.
Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement...
There are lots of Stratum 0 servers out there; basically anything with an atomic clock will do. They all count seconds independently from one another, all slowly diverging over time, with offset intervals being measured by mutual synchronization using a number of means (how is this done is interesting all by itself). Some atomic clocks are more accurate than others, and an ensemble of these is typically regarded as 'the' master clock.
Beyond this, as other commenters have said, anyone who is really dependent on having exact time (such as telcos, broadcasters, and those running global synchronized databases) should have their own atomic clock fleets. There are thousands and thousands of atomic clocks in these fleets worldwide. Moreover, GPS time, used by many to act as their time reference, is distributed by yet other means.
Nothing bad will happen, except to those who have deliberately made these specific Stratum 0 clocks their only reference time. Anyone who has either left their computer at its factory settings or has set up their NTP configuration in accordance to recommended settings will be unaffected by this.
Residents and some businesses of Boulder have been without power since Tuesday. There was an issue about 10 years ago which caused 1000 homes to burn down and the power company was found liable. They change their actions. Then during the next high wind event, the power company preemptively cut power and businesses sued them for loss of revenue. Now the power company is playing it safe and turning off power to residents and keeping downtown businesses powered.
Maybe their generator failing was DOGE related, but wouldn’t have happened if state level shenanigans were better handled
I just got home from a holiday party where two people from NIST in Boulder work on the NIST-F1 Atomic Clock. The one scientist described the outage and the failures but reported that things are still shut down even after power was restored. She told me that she believes that the Trump administration is in the process of shutting down the program and server in Boulder. And there was low confidence in that the Washington DC superiors understood what they are doing. God I hope this isn't more Tina Peters retribution crap. Cutting off our nose to spite our face.
It’s fine. The public pays for sci-fi clocks used by NIST and the Navy and we get shit latency over NTP and a WWVB signal that barely reaches a huge chunk of the country. CLOCKS WE PAID FOR. Jane Street gets lightning access to clocks and our pension managers get their NTP trades front run. NTP is a disgrace and an insult when it is working.
Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious. A lot of power was shut off preemptively to prevent downed power lines from starting wildfires. Energy providers gave warning to locals in advance. Shame that NIST’s backup generator failed, though.
Notably, we had the marshal fire here 4 years ago and recently Xcel settled for $680M for their role in the fire. So they're probably pretty keen not to be on the hook again
I guess that explains why they had no qualms shutting down half of Boulder's power with a vague time horizon. After losing everything in my fridge, though, they finally turned it back on today.
4 replies →
For more background on the Marshal Fire of Dec. 2021: https://en.wikipedia.org/wiki/Marshall_Fire
tl;dr - the fire destroyed over 1,000 homes, two deaths. The local electrical utility, Xcel, was found as a contributing cause from sparking power lines during a strong wind storm. As a result, electrical utilities now cut power to affected areas during strong winds.
Somewhat interesting that they themselves don't have access to the site. You'd think there would have been some disaster plans put in place?
The disater plan is to have a few dozens stratum 1 servers spread around the world, each connected to a distinct primary atomic clock, so that a catastrophic disaster needs to take down the global internet itself for all servers to become unreachable.
The failure of a single such server is far from a disaster.
12 replies →
If access to the site is unsafe and thus the site is closed; not having access seems reasonable.
Time services are available from other locations. That's the disaster plan. I'm sure there will be some negative consequences from this downtime, especially if all the Boulder reference time sources lose power, but disaster plans mitigate negative consequences, they can't eliminate them.
Utility power fails, automatic transfer switches fail, backup generators fail, building fires happen, etc. Sometimes the system has to be shut down.
1 reply →
Maybe this is the disaster plan: There's not a smouldering hole where NIST's Boulder facility used to be, and it will be operational again soon enough.
There's no present need for important hard-to-replace sciencey-dudes to go into the shop (which is probably both cold, and dark, and may have other problems that make it unsafe: it's deliberately closed) to futz around with the the time machines.
We still have other NTP clocks. Spooky-accurate clocks that the public can get to, even, like just up the road at NIST in Fort Collins (where WWVB lives, and which is currently up), and in Maryland.
This is just one set.
And beyond that, we've also got clocks in GPS satellites orbiting, and a whole world of low-stratum NTP servers that distribute that time on the network. (I have one such GPS-backed NTP server on the shelf behind me; there's not much to it.)
And the orbital GPS clocks are controlled by the US Navy, not NIST.
So there's redundancy in distribution, and also control, and some of the clocks aren't even on the Earth.
Some people may be bit by this if their systems rely on only one NTP server, or only on the subset of them that are down.
And if we're following section 3.2 of RFC 8633 and using multiple diverse NTP sources for our important stuff, then this event (while certainly interesting!) is not presently an issue at all.
14 replies →
Step One of most disaster plans is not to create a second emergency.
13 replies →
> Wind gusts were reaching 125 MPH in Boulder county, if anyone’s curious.
That's some strong winds! What's causing such strong sustained/gusty winds that long? I'm hearing about this weather phenomenon for the first time.
Yup, here in Jefferson County - roughly 30 minutes south of Boulder County, we were getting wind gusts around 80mph.
Of the various internet .+P, NTP is one I never learned about as a student, so now I'm looking at its web page [1] by its creator David L. Mills (1938-2024). I've found one video of him giving a retrospective of his extensive internet work; he talks about NTP at 34:51 [2] and later at 56:26 [3].
[1] https://www.eecis.udel.edu/~mills/ntp.html
[2] https://youtu.be/08jBmCvxkv4?si=WXJCV_v0qlZQK3m4&t=2092
[3] https://youtu.be/08jBmCvxkv4?si=K80ThtYZWcOAxUga&t=3386
HN discussion shortly after Dave Mills died, early in 2024: https://news.ycombinator.com/item?id=39051246
Amazingly, The New Yorker wrote an article about it: https://www.newyorker.com/tech/annals-of-technology/the-thor...
In [3] he mentions that one can use NTP to observe frequency deviations and use it as an early warning system for fire and AC failure. That really intrigues me. Can you actually? Has this ever been implemented?
Oscillators of all kinds are temperature dependent.
That's why the most stable ones are insulated and ovenized[1].
So an AC failure which would lead to higher room temperatures would lead to stronger or more frequent correction by the NTP client, as the local oscillator would drift more.
Not sure about the fire case though. I mean the same applies there but I'm not imaginative enough to think of a realistic scenario where NTP would be useful for averting a fire.
[1]: https://blog.bliley.com/anatomy-of-an-ocxo-oven-controlled-c...
I knew of some experiments in this space back in the late 1980s or early 1990s - but it was specifically with DECstation hardware that had terrible clocks (not used for alerting, just "this graphs nicely against temperature".) https://groups.csail.mit.edu/ana/Publications/PubPDFs/Greg.T... (PDF) 4.2.1 does talk about explaining local clock frequency changes with office temperature changes (because they overwhelm a clock-aging model) but it doesn't have graphs so perhaps they weren't clear enough to include (or just not relevant enough to Time Surveying.)
What do you mean by +P?
Presumably, .+P is regex referring too all the acronyms ending with P, that is Protocol. SMTP, HTTP, FTP, IMAP, XMPP…
Looks like .+P is just a regex way of saying any set of characters / protocol ending in P.
Nothing, as they said .+P (note the dot)
> Facility operators anticipated needing to shutdown the heat-exchange infrastructure providing air cooling to many parts of the building, including some internal networking closets. As a result, many of these too were preemptively shutdown with the result that our group lacks much of the monitoring and control capabilities we ordinarily have
Having a parallel low bandwidth, low power, low waste heat network infrastructure for this suddenly seems useful.
Like reticulum?
https://github.com/markqvist/Reticulum/
NIST campus status: Due to elevated fire risk and a power outage for the Boulder area, the DOC Boulder Labs campus is CLOSED on December 19 for onsite business and no public access is permitted; previously approved accesses are revoked.[1]
WWV still seems to be up, including voice phone access.
NIST Boulder has a recorded phone number for site status, and it says that as of December 20, the site is closed with no access.
NIST's main web site says they put status info on various social media accounts, but there's no announcement about this.
[1] https://www.nist.gov/campus-status
Note that WWV is a good 60 miles NNE of Boulder on the outskirts of Fort Collins.
Can anybody expand on the implications of this?
Being unfamiliar with it, it's hard to tell if this is a minor blip that happens all the time, or if it's potentially a major issue that could cause cascading errors equal to the hype of Y2K.
Time travel is extremely dangerous right now. I highly recommend deferring time travel plans except for extreme temporal emergencies.
Same for database transaction roll back and roll forward actions.
And most enterprises, including banks, use databases.
So by bad luck, you may get a couple of transactions reversed in order of time, such as a $20 debit incorrectly happening before a $10 credit, when your bank balance was only $10 prior to both those transactions. So your balance temporarily goes negative.
Now imagine if all those amounts were ten thousand times higher ...
2 replies →
Uhh, here's the problem, I'm sort of stuck travelling into the future at a more or less constant rate. I don't know how to stop doing that...
7 replies →
Unless you want to punch a guy in the face offscreen. Then go for it.
Would traveling to the past in order to put in place a preemptive fix for this outage be wise or dangerous?
Asking for a friend.
7 replies →
Define “extreme”?
Google has their own fleet of atomic clocks and time servers. So does AWS. So does Microsoft. So does Ubuntu. They're not going to drift enough for months to cause trouble. So the Internet can ride through this, mostly.
The main problem will be services that assume at least one of the NIST time servers is up. Somewhere, there's going to be something that won't work right when all the NIST NTP servers are down. But what?
Ubuntu using atomic clocks would surprise me. Sure they could, but it's not obvious to me why they would spend $$$$ on such. More plausible to me seems that they would be using GPSDO as reference clocks (in this context, about as good as your own atomic clock), iff they were running their own time servers. Google finds only that they are using servers from the NTP Pool Project, which will be using a variety of reference clocks.
If you have information on what they actually are using internally, please share.
7 replies →
Atomic clock non-expert here, what does having a fleet of atomic clocks entail and why would the hyperscalers bother?
10 replies →
Can't they point these dns records to working servers meanwhile to avoid degradation?
2 replies →
I know this is HN, but the internet is pretty low on the list of things NIST time standards are important for.
19 replies →
GPS?
NIST maintains several time standards. Gaithersburg MD is still up and I assume Hawaii is as well. Other than potential damage to equipment from loss of power (turbo molecular vacuum pumps and oil diffusion pumps might end up failing in interesting ways if not shut down properly) it will just take some time for the clocks to be recalibrated against the other NIST standards.
Gaithersburg went down on December 8th, is there confirmation that it's fully functional again?
1 reply →
Time engineers are very paranoid. I expect large problems can't occur due to a single provider misbehaving.
No noteworthy impact at all. The NTP network has hundreds to thousands of redundant servers and hundreds of redundant reference clocks.
The network will route around the damage with no real effects. Maybe a few microseconds of jitter as you have to ask a more distant server for the time.
>Can anybody expand on the implications of this?
The answer is no. Anyone claiming this will have an impact on infrastructure has no evidence backing it up. Table top exercises at best.
Songs with lyrics such as, "What time is it?" will have no clap back.
Perhaps, "We don't know." will become popular?
If your computer was using it as your time server and you didn't have alternatives configured your clock my have drifted a few seconds.
I never checked it, but how much a typical's pc/server's clock does actually drift over a week or a month? I always thought it's well under a second.
5 replies →
Unix timestamp resets to zero.
I always wanted to see what the 70s were like, heres to hoping it happens.
I’d say everybody moving off NIST boulder NTP.
FYI, this was posted a month ago when discussing thermal effects of clock drift. I thought it was quite interesting view of what the WWVB location looks like:
https://news.ycombinator.com/item?id=46042946
Status of NIST time servers:
https://tf.nist.gov/tf-cgi/servers.cgi
Old sci-fi story of mine, maybe the only one ever inspired by NTP
THE TIME RIFT OF 2100: How We lost the Future --- and Gained the Past.
https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
Cool story - thanks for sharing it!
This makes me wonder, if you take the average time of all wristwatches on the planet, accounting for timezones and throwing out outliers, how close would you get to NTP time?
And how many randomly chosen wristwatches would you need to get anything reasonable?
You’re the person Douglas Adams warned us about.
Comment of the year.
I have a hunch my casio wrist watch is designed to be running a bit too quick to make resetting the seconds easier. Your averaging assumes manufacturers try to make their watches as accurate as possible for average conditions
I think it runs quick to be on the safe side, so you never miss appointments, trains, etc. because of your watch.
But yes, good point.
This is the kind of thing that Casio designers would probably come up with (second to have as much accuracy as possible within their budget)
Given two time changes per year I guess something like 1 min per year is acceptable
Close but unlikely to be precise in metrology sense. There's unlikely even a billion wrist watches being worn.
One. One watch. POTUS's watch. And in fact, that's why Boulder is currently shuttered... they disagreed.
I think this comment is referencing the government's recent announcement[0] to shut down the National Center for Atmospheric Research in Boulder. They do climate research at the Mesa Laboratory there.
It's open to the public for visits. They have a small science museum, offices, a library, etc. I highly recommend anyone with interest and opportunity to visit the mesa lab soon. It may not be open much longer. The view alone is worth the trip, and the building is cool too.
[0]: https://x.com/russvought/status/2001099488774033692
1 reply →
We had some fun requesting key for accessing nist time servers. the process is (quoted from website)
NIST currently offers this service free of charge. We require written requests to arrive by U.S. mail or fax containing:
Your organization’s name, physical address, fax number (if desired as a reply method).
One or more point-of-contact personnel or system operators authorized to receive key data and other correspondence: names, phone numbers, email addresses. Up to four static IPv4 network addresses under the user’s control which will be allowed to use the unique key. By special arrangement, additional addresses or address ranges may be requested.
Desired hash function (“key type”). NIST currently supports MD5, SHA1, SHA256, and HMAC-SHA256. Please list any limitations your client software places on key values, if known: maximum length, characters used, or whether hexadecimal key representations are required. If you prefer, please share details about your client software or NTP appliance so we can anticipate key format issues. Desired method for NIST’s reply: U.S. mail, fax, or a secure download service operated by Department of Commerce.
NIST will not use email for sending key data.
ps. there actually seems to be improvement over what they had year ago. they added "secure download service". and previously they had message that nobody assigned to actively monitor mailbox so if you didn't get key, please email us so we will check it
Can anybody speak to the current best practices around running underground power lines? I see these types of articles about above-ground distribution systems from time-to-time, particularly in California. I feel lucky that my area has underground power, but that was installed back in the 1980s. Would it be prohibitively expensive for Boulder’s utility provider to move to underground distribution? I can’t help but think it could be worth the cost to reduce wildfire risk and offer more reliable service.
Think of it like this: overhead power lines require you to dig a 5-7’ deep hole that’s 2’ in diameter every 90’. Underground power supplied through cable requires you to bury the cable minimum 3’ in the ground in rigid ductwork the entire 90’. Any time that cable runs under a roadway that ductwork needs to be encased in concrete. In urban and semi urban areas you also compete with other buried infrastructure for space - sewer, city/municipal infrastructure, gas, electrical transmission, etc.
While underground distribution systems are less prone to interruption from bad weather it depends on the circuit design. If the underground portion of the circuit is fed from overhead power lines coming from the distribution substation you will still experience interruptions from faults on the overhead. These faults can also occur on overhead transmission circuits (the lines feeding the distribution substations and/or very large industrial customers).
Underground distribution comes at a cost premium compared to overhead distribution. It’s akin to the cost of building a picket fence vs installing a geothermal heating system for your home. This is why new sub divisions will commonly have underground cable installed as the entire neighborhood is being constructed - there’s no need to retrofit underground cable into an existing area and so the costs are lower and borne upfront.
It’s more cost effective for them to turn the power off as a storm rolls through, patrol, make repairs and reenergize then to move everything underground. Lost revenue during that period is a small fraction of the cost of taking an existing grid and rebuilding it underground. This is especially true for transmission circuits that are strung between steel towers over enormous distances.
Boulder County is working on utility undergrounding, https://bouldercolorado.gov/services/utility-undergrounding, and in particular they are working on undergrounding at Chautauqua, https://bouldercolorado.gov/news/undergrounding-chautauqua, which is next door to the NIST facility.
1/3-1/2 of the cost of the electricity we pay is distribution.
Some of it is physical infrastructure (transformers, wire, poles), but a lot of it is labor.
Labor is expensive in US. It’s a lot of labor to do, plus they’ll likely need regulatory approval, buying out land, working through easements.
At the same time you have people screaming about how expensive energy is.
Furthermore they have higher priorities, replacing ancient aging infrastructure that’s crumbling and being put on higher load every day.
Its very hard to repair and keep track of underground.
https://practical.engineering/blog/2021/9/16/repairing-under...
Germany here, never heard of any issues regarding underground power (or phone) lines. Ultra High voltage (distribution network) is above ground here, but no issued with that either.
This was an NTP 0 server right? What is the actual failback mechanism when that level of NTP server fails?
This is some level of eldritch magic that I am aware of, but not familiar with but am interested in learning.
There's two other sites for the time.nist.gov service so it'll be okay.
Probably more interesting is how you get a tier 0 site back in sync - NIST rents out these cyberpunk looking units you can use to get your local frequency standards up to scratch for ~$700/month https://www.nist.gov/programs-projects/frequency-measurement...
What happens in the event all the sites for time.nist.gov go down? is it included in the spec?
Also thank you for that link, this is exactly the kind of esoteric knowledge that I enjoy learning about
2 replies →
Considering how many servers are in existence, probably the exact same procedure for starting a brand new one?
I must have one of those units oh my god
2 replies →
There are lots of Stratum 0 servers out there; basically anything with an atomic clock will do. They all count seconds independently from one another, all slowly diverging over time, with offset intervals being measured by mutual synchronization using a number of means (how is this done is interesting all by itself). Some atomic clocks are more accurate than others, and an ensemble of these is typically regarded as 'the' master clock.
To quote the ITU: "UTC is based on about 450 atomic clocks, which are maintained in 85 national time laboratories around the world." https://www.itu.int/hub/2023/07/coordinated-universal-time-a...
Beyond this, as other commenters have said, anyone who is really dependent on having exact time (such as telcos, broadcasters, and those running global synchronized databases) should have their own atomic clock fleets. There are thousands and thousands of atomic clocks in these fleets worldwide. Moreover, GPS time, used by many to act as their time reference, is distributed by yet other means.
Nothing bad will happen, except to those who have deliberately made these specific Stratum 0 clocks their only reference time. Anyone who has either left their computer at its factory settings or has set up their NTP configuration in accordance to recommended settings will be unaffected by this.
The referenced mailing list is this Google Group (https://groups.google.com/a/list.nist.gov/g/internet-time-se...) which has some other posts about this incident.
Where are you seeing the other posts?
So far I think I'm still seeing one of them in my peers list for my public-ish NTP server:
...and maybe it's gone:
One question I have is did DOGE decisions have anything to do with this? Because I know they took knives to NIST.
Residents and some businesses of Boulder have been without power since Tuesday. There was an issue about 10 years ago which caused 1000 homes to burn down and the power company was found liable. They change their actions. Then during the next high wind event, the power company preemptively cut power and businesses sued them for loss of revenue. Now the power company is playing it safe and turning off power to residents and keeping downtown businesses powered.
Maybe their generator failing was DOGE related, but wouldn’t have happened if state level shenanigans were better handled
The Marshall fire was 4 years ago. Almost to the day.
Actually DOGE involvement at the highest level would have resulted in Tesla solar and Tesla powerwall battery backups.
> […] Tesla solar and Tesla powerwall battery backups.
Don't forget Solar Roof.
Apparently no engineers here anymore, all armchair politicians.
2 replies →
Lol, lmao.
Some relevant DOGE’s effects:
-time and frequency division director quit
-NIST emergency management staff at least 50% vacant
-NIST director of safety retired, and NIST safety was already understaffed when compared to DOE labs
-NOAA emergency manager on the same Boulder campus laid off
etc
Any day now
Direct link, rather than a repost: https://groups.google.com/a/list.nist.gov/g/internet-time-se...
My local GPS + PPS based NTP server finally pays off.
It’d be a good idea to protect our infrastructure from the climate we created.
It’s just a good idea, though, not a greedy one… so it won’t happen.
I just got home from a holiday party where two people from NIST in Boulder work on the NIST-F1 Atomic Clock. The one scientist described the outage and the failures but reported that things are still shut down even after power was restored. She told me that she believes that the Trump administration is in the process of shutting down the program and server in Boulder. And there was low confidence in that the Washington DC superiors understood what they are doing. God I hope this isn't more Tina Peters retribution crap. Cutting off our nose to spite our face.
Man, they're having a hell of a time up in Boulder.
Generator maintenance is a thing.
This is terrible. It’s always sad to hear about things like this.
Well, where did NTP at NIST last put it? Did they look there?
You misunderstand, there's been a coup
Of course there is. Where else would they put the reference standard chickens?
We have to stop those knaves pushing PTP! NTP must prevail!
For future reference of civilization: if a facility is critical, it must have a SMR.
It’s fine. The public pays for sci-fi clocks used by NIST and the Navy and we get shit latency over NTP and a WWVB signal that barely reaches a huge chunk of the country. CLOCKS WE PAID FOR. Jane Street gets lightning access to clocks and our pension managers get their NTP trades front run. NTP is a disgrace and an insult when it is working.
The rest of the snark/ bitterness aside ... NTP takes latency as a constraint, and accommodates for it.
And far from a disgrace, NTP was a brilliant design for its time, and has proven to be far more future-proof than should have been expected.