Comment by Johnny555
5 years ago
But don't their datacenters all have backup generators? So worst case in a brownout, they fail over to generator power, then can start to flip back to utility power slowly.
Or do they forgo backup generators and count on shifting traffic to a new datacenter if there's a regional power outage?
Edit to be less snarky:
I assume they do have backup generators, though I don’t know.
However if the sudden increase put that much load on the grid it could drop the frequency enough to blackout the entire neighborhood. That would be bad even if FB was able to keep running through it.
Ah yea I meant brownouts for other people haha. I figure Facebook can handle their own electrical stability just fine
The blackout of the northeast US and parts of Canada, in 2003 was really caused by something relatively small. Imagine Facebook, yesterday, causing some weird cascading effect on the power grid, and pulling half of the country with it...
Is there any liability if Facebook had brought everything up at once and caused brownouts? Seems like it would be some form of negligence on their part harming a shared resource, but I don't know if there's any laws or contract terms with the power company that require them to pay if they mess up like that.
My girlfriend works in a large grid operator (in Europe). According to her there are lots of regulations and contracts on the grid operators about how they must handle reliability. So it's unlikely that Facebook would be liable if this took down half the country, because then it was the grid operator not living up to their agreements on reliability.
There are a lot of automated fail-safes on this, and apparently larger industry (which a datacenter is as well) will get disconnected from the grid automatically in emergency situations before they drop residential areas. But in the end they will drop one by one everything they need to keep the larger grid running. It's not even a networked "smart" management system, the distribution points automatically react to voltage and frequency drops and they're set up to break some things like industry earlier than others.
For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly or are constantly running to provide a buffer.
Short term they'd help but an instantaneous or unexpected massive traffic/CPU usage/user surge might pop too fast for the generators to start and kick in properly. Also, it might not be good for those big generators to start and stop over and over vs bringing infra back online in waves to limit spikes.
For outages the generatos are great but I'm not sure how they assist with brownouts unless they can start instantly
If the generators will help in an outage, why wouldn't they help in a brownout? You'd transition to generator when the voltage and/or frequency is outside of spec.
You'd typically have some short-term power protection to keep your datacenter running until the generators start.
I was skeptical about a datacenter that had less than 60 seconds of flywheel energy storage. But the data center manager said that if the generator doesn't start within 30 seconds, you're not going to get it started in an hour so having a huge battery stack that can power the datacenter for 15 minutes isn't going to help much.
Generators do usually start up very quickly. Under a minute.
I guess a DC would have UPS/battery power on hand to cover an instantaneous brown out. Then the generators could be on and restoring battery power while running the DC.