Comment by dan_sbl
2 hours ago
> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power.
I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?
If my gpu is sitting idle, and I mean idle with nothing loaded into its memory, it's sitting at about 18W. If I load in model that uses nearly all of the memory but that model is idle, it's at 36W. If that model is actively thinking, it's like 118W. I think this is likely due to the GPU being aware that there is real data loaded into memory and turning up the DRAM refresh rate whereas when nothing is loaded, the dynamic power is as low as possible.
Server cards are not optimized for idle power usage. They’re expected to be fully utilized.
For server gear it’s more common to have less dynamic power and voltage switching because it produces more predictable performance and latency.
For GeForce cards you can get similar behavior by setting “Prefer maximum performance” which disables some of the low power states.
I suspect the act of running nvidia-smi itself prevents the GPU from being put into a low-power state.