← Back to context

Comment by threwrfaway

8 hours ago

For those who dont know why this is important:

The loads are slowing down the generators that are burning a well metered amount of fuel to stay at 60Hz. This is a delicate balance since the phase angle must also be spot on.

If a generator and the local line disagree on f, phase or V, you have a short circuit.

If you lose a large amount of load, your generator will spin up with the excess fuel until the control system re-establishes the right amount of fuel.

But now your generators are out of sync! No worry, for small disturbances the dissipative losses sync everything up like syncros on a manual transmission.

But the disturance cant be too big!

Rotating machines are big and heavy, so the first line of defense is their inertia. But this is a finite (and precious) resource.

Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse. They generate large amounts of power while providing no inertia.

You'd think it isn't a big deal since the DC-AC converter can just synthesize whatever is needed. Heck just keep it rigid at 60 Hz with no phase change.

Well the later doesn't work - the rest of the grid is no longer at that phase and frequency so you got yourself a short.

Furthermore, the DC-AC converter, despite their manufacturers' promise, has no good way to establish what f and phase it should be at during a disturbance (and these magic codes are closed source, believe it or not)

Anywho, a large enough loss of load causes the grid to enters into unstable oscillations, causing protective relays to trip causing a zipper effect where the grid goes down.

Now restart will take a few days depending on the energy mix (fastest for hydro heavy)

Long story short - this is not a trivial problem, and the data-centers can't be allowed to just dump load willy nilly.

EDIT: made it clear that the grid killing disturbance is not caused by renewables; not exclusively anyway. Everyone has to play nice or the grid goes down.

Grid forming batteries and inverters are a thing. They can control the frequency on the grid. Just a matter of getting the right equipment.

The nice thing with data centers is that they are somewhat flexible. It's not a constant load. Data center operators can choose to reduce load. And if properly engineered, they could do so automatically based on signals from the grid.

The issue with outdated grids is that it relies on technology (spinning mass) that's at this point a century old. Which makes it brittle against outages like you describe. The solution is not more spinning mass but batteries and renewables to take the place of that spinning mass. A battery can respond to oscillations in milliseconds. If you then add flexible load that can spin up/down based on the amount of available power, you gain a lot of stability.

  • Replying to a description of a problem in a highly complex system with a solution that begins with the word "just" does nothing for your credibility.

  • > The issue with outdated grids is that it relies on technology (spinning mass) that's at this point a century old. Which makes it brittle against outages like you describe. The solution is not more spinning mass but batteries and renewables to take the place of that spinning mass. A battery can respond to oscillations in milliseconds. If you then add flexible load that can spin up/down based on the amount of available power, you gain a lot of stability.

    Why the spinning mass technology being "century old" (more like "millennia old" but anyways) is a problem somehow? The Newton's First Law didn't change much in the time that has passed.

    Anyone who tried to "respond to oscillations in milliseconds" knows how hard that problem is because the force you apply is integrated twice before it takes effect. Try stabilizing a swing by pushing it's forward when it's behind the equilibrium point and pushing it backward when it's ahead of it. Now imagine a grid of swings connected by rubber bands and a distributed system of independent actors responding to oscillations. There are much more ways in which this system can diverge rather then converge.

    Time may prove me wrong, but the arguments like "spinning mass is old therefore should be replaced" certainly won't.

    • > Newton's First Law didn't change much in the time that has passed.

      No, but technology has moved on quite a bit. Heavy fly wheels are no longer the state of the art here.

      A few tens of GW of battery capacity (i.e. a few dozen nuclear plants worth of capacity) that can switch on/off in milliseconds can do a lot for grid stability. That's part of the reason why grid operators are rolling out so much batteries. It's not necessarily about supplying energy for a very long time but about smoothing out peaks and dips in energy supply and demand and responding more or less in real time to that.

      This stuff is basically being rolled out at industrial scale in a lot of places. Australia, China, etc. pretty much run increasingly on mostly renewables. This is no longer as speculative as it would have been ten years ago.

      Yes, there are engineering challenges with rolling that stuff out in a lot of places. And even more policy and regulation challenges. Actually that is, by far, the #1 challenge in places like the US and Europe. Grid operators are simply structured and incentivized wrong to deal with this stuff efficiently. Texas is actually not doing too bad relative to e.g. California. But they clearly have some challenges still.

  • Yes we know the solution is infrastructure. The question is who gets to pay for it.

    In America that seems to be the dying small town whose only economic value is cheap land.

That’s a simplified and somewhat outdated version, there’s a huge range of technology to mitigate each of these issues but infrastructure generally isn’t free. A turbine does provides inertia for free where you need to pay for a flywheel or battery system.

In the end a renewable heavy grid can be extremely resilient due to all the batteries, but smart battery systems need to be incentivize or mandated because ‘dumb’ batteries are cheaper.

10+ MW voltage-source converters that can't do up to around 80% of their nominal capacity as mostly-reactive apparent power with stabilizing synthetic inertia scaled as desired/specified are a mostly software issue, stemming from lack of regulatory pressure incentivizing the engineering complexity of that.

Though if you want to do a smoothing action on real power flux you'll have to colocate battery capacity with the converter. Which to be clear is fairly cheap to do as long as you get compensated for the substantial frequency stabilization capacity this represents. I'm talking like 15~120 minutes at converter nominal AC power of battery capacity.

The first 10~20% of reactive power are almost free from the converter electronics, btw....

The ELI5 version is you're arm-wrestling someone and they suddenly let their arm go limp, so your arm slams down on the table since you can't react that fast.

  • To build on this analogy, tug of war fits a bit better. Nothing dramatic happens if one person let go, but if half of one team just let loose at the same time without communication, bad things happen.

I've often thought that the target* grid phase should be encoded into a high frequency signal, say at 50kHz. Generators without inertia can immediately switch to the required phase and inertial systems can work towards it or disconnect their outputs if they stray too far.

The problem in my mind is that, as it stands, the signal that everyone has to latch on to is itself affected by load and by different generators latching on with different time constants in a complicated feedback loop. Would having a single authoritative source be an improvement? Would that be a way to eliminate the need for inertia?

*I suppose the "target" phase would probably be based primarily on the output from the biggest inertial systems and take into account their ability to adjust phase and frequency.

Would demanding that large spikey users of energy like data centers implement some sort of demand ramping/isolation from the grid in the form of a massive capacitor bank or flywheel generator between them and the grid help reduce the risk here?

  • The data centres aren't inherently spikey, in general use their consumption is reasonably predictable.

    However, if a DC detects that the _grid_ is wobbly (voltage or frequency deviations) the DC will disconnect without warning, and switch to its batteries and generators.

    The grid complains because it's suddenly lost hundreds of MW of load. For the DC to have isolation capability, it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice.

    It's a hard problem to solve, and probably better managed at grid-level than DC-level.

    • > The grid complains because it's suddenly lost hundreds of MW of load. For the DC to have isolation capability, it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice.

      Thats why always have all the EVs be connected all the time (except the 20 mins they drive). EVs can provide demand as a service and take excess power whenever its available (instead of solar curtailment), and also provide an immediate source of load when events like this happen. Its a shame US is anti EV, it has the best systems at scale that can be leveraged to transform the entire energy ecosystem.

    • My intuition is that there would be a fairly stable base load, but doing something like switching on a new training run of a frontier model would be incredibly spiky, thousands of GPUs going from somewhat idle to 100% in seconds.

    • >it would need a load-sink which can consume roughly the same power as the DC in normal operation, and can take in that load at a moment's notice.

      Liquid rheostat. A big one.

Some of us in Texas are all too familiar with the problem of balancing load with generation, the risk of a cascade failure causing a slow restart.

During winter storm Uri, they did a duty cycle where we only had power available for ~6-12 hours at a time on the days it was available. This was apparently to avoid that very problem.

So far as I know, the obvious mitigations like winterizing NG generation and/or peering with neighbor grids have not been performed.

> Contrary to belief, renewables, or generally speaking DC, makes things this stability problem worse.

Is there such belief? My feel is that anybody to whom electrical grid stability has even crossed their minds know this.

When you say "short circuit" - is this when wires get hit and make and burn, or this is reference to some other stuff?

  • Short circuit is when two sources try to drive the same node in a circuit at different voltages or frequencies. In an idealized circuit (such as where the wires are assumed to have zero resistance), current flow from one source to the other is infinite. In a real circuit, current flow rises to a level where "something has to give." This could include heating the wires between the sources or tripping some kind of protection mechanism.

    I'm not great with analogies, but imagine a train with two locomotives, and each one is set to run at a different speed. There will probably be a lot of screeching of wheels etc. To make multiple locomotives work on a single train requires engineering them to be synchronized with one another.

Is it feasible to tie the grid frequency to an external clock reference, such as TAI?

So apparently grid forming or synthetic inertia can be provided electronically nowadays, but presumably it would not be too terrible to put a rotating mass in between the (e.g.) solar power source and the rest of the grid? So the electricity from the solar panels runs a motor that turns a generator. I suppose power is lost through friction and other inefficiencies, and you'd need to co-locate the solar panels into a big enough farm.