Comment by coldtea
12 hours ago
Would it be that many? Asked AI to do some rough calculation, and it spit that:
Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day
Given I usually have 2-3 lights on all day in the house, that's like 1500 LLM requests per day (which sounds quite more than I do).
So even a month worth of requests for building some software doesn't sound that much. Having a local beefy traditional build server compling or running tests for 4 hours a day would be like ~7,600 requests/day
> Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day
This seems remarkably far from what we know. I mean, just to run the data centre aircon will be an order of magnitude greater than that.
Is that true? Because that's indeed FAR less than I thought. That would definitely make me worry a lot less about energy consumption (not that I would go and consume more but not feeling guilty I guess).
A H100 uses about 1000W including networking gear and can generate 80-150 t/s for a 70B model like llama.
So back of the napkin, for a decently sized 1000 token response you’re talking about 8s/3600s*1000 = 2wh which even in California is about $0.001 of electricity.
With batched parallel requests this scales down further. Even a MacBook M3 on battery power can do inference quickly and efficiently. Large scale training is the power hog.