Comment by plorg

5 years ago

Brownouts is probably the most proximate concern - a sudden increase in demand will draw down the system frequency in the vicinity, and if there aren't generation units close enough or with enough dispatchable capacity there's a small chance they would trip a protective breaker.

A person I know on the power grid side said at one data center there were step functions when FB went down and then when it came up, equal to about 20% of the load behind the distribution transformer. That quantity is about as much as an aluminum smelter switching on or off.

13 comments

plorg

Reply
bigiain  5 years ago

> That quantity is about as much as an aluminum smelter switching on or off.

Interestingly, the mountains east of Portland OR, where all the Aluminum smelters used to be, are now full of FAANG datacenters relying on the power infrastructure (and pricing) the Aluminum industry used to use...

https://www.oregonlive.com/silicon-forest/2015/10/small-town...

And Washington state too:

https://www.bizjournals.com/seattle/blog/techflash/2015/11/p...

  • Cthulhu_  5 years ago

    That's pretty interesting, I'm sure those aluminium foundries would need to be careful about turning the power on as well.

    Tangentially related, aluminium production in the Netherlands may shut down soon; because of a sudden spike in gas prices (due to mismanagement), electricity prices have also gone up, making producing aluminium no longer cost-effective. €2400 in electricity to produce a ton of aluminium worth €2500 kinda cost effectiveness.

    I wouldn't be surprised if the big datacenters here will try and offload some of their workloads to datacenters elsewhere with lower energy costs. Mind you, I'm pretty sure these datacenters make long-running deals on electricity prices.

Johnny555  5 years ago

But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly.

Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?

  • kgermino  5 years ago

    Edit to be less snarky:

    I assume they do have backup generators, though I don’t know.

    However if the sudden increase put that much load on the grid it could drop the frequency enough to blackout the entire neighborhood. That would be bad even if FB was able to keep running through it.

  • dmoy  5 years ago

    Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine

    • sverhagen  5 years ago

      The blackout of the northeast US and parts of Canada, in 2003 was really caused by something relatively small. Imagine Facebook, yesterday, causing some weird cascading effect on the power grid, and pulling half of the country with it...

    • ziml77  5 years ago

      Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.

      1 reply →

  • charwalker  5 years ago

    For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer.

    Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringing infra back online in waves to limit spikes.

    • Johnny555  5 years ago

      For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly

      If the generators will help in an outage, why wouldn't they help in a brownout? You'd transition to generator when the voltage and/or frequency is outside of spec.

      You'd typically have some short-term power protection to keep your datacenter running until the generators start.

      I was skeptical about a datacenter that had less than 60 seconds of flywheel energy storage. But the data center manager said that if the generator doesn't start within 30 seconds, you're not going to get it started in an hour so having a huge battery stack that can power the datacenter for 15 minutes isn't going to help much.