Comment by lpcvoid
7 hours ago
> Open source efforts need to give up on local AI and embrace cloud compute.
Oh god no, please not more slop, you're already consuming over 1 percent of human energy output, could you, like, chill a bit?
7 hours ago
> Open source efforts need to give up on local AI and embrace cloud compute.
Oh god no, please not more slop, you're already consuming over 1 percent of human energy output, could you, like, chill a bit?
In a similar vein: seek efficiency.
I.e., /if/ I am going to consume LLM tokens, I figure that a local LLM with 10s of billions of parameters running on commodity hardware at home will still consume far more energy per token than that of a frontier model running on commercial hardware which is very strongly incentivized to be as efficient as possible. Do the math; it isn't even close. (Maybe it'd be closer in your local winter, where your compute heat could offset your heating requirements. But that gets harder to quantify.)
Maybe it's different if you have insane and modern local hardware, but at least in my situation that is not the case.
But commodity hardware that's right-sized for your own private needs is many orders of magnitude cheaper than datacenter hardware that's intended to serve millions of users simultaneously while consuming gigawatts in power. You're mostly paying for that hardware when you buy LLM tokens, not just for power efficiency. And your own hardware stays available for non-AI related needs, while paying for these tokens would require you to address these needs separately in some way.
>And your own hardware stays available for non-AI related needs, while paying for these tokens would require you to address these needs separately in some way.
^ Fair. Yep, I agree the calculus changes if you don't have _any_ local hardware and you're needing to factor in the cost of acquiring such hardware.
When I did this napkin math, I was mostly interested in the energy aspect, using cost as a proxy. I was calculating the $/token (taking into consideration the cost of a KWh from my utility, the measured power draw of my M1 work machine, and the measured tokens per second processed by a ~20BP open-weight model). I then compared this to the published $/token rate of a frontier provider, and it was something like two orders of magnitude in favor of the frontier model. I get it, they're subsidizing, but I've got to imagine there's some truth in the numbers.
I wonder, does (or will) the $/token ratio fall asymptotically toward the cost of electricity? In my mind I'm drawing a parallel to how the value of mined cryptocurrency approximately tracks the cost of electricity... but I might be misremembering that detail.
I doubt it because you aren't going to get the utilisation that a commercial setup would. No point wasting tons of money on hardware that is sat idle most of the time.
1 reply →
Y'all aren't seeing the same future I am, I guess.
- Our career is reaching the end of the line
- 99.9999% of users will be using the cloud
- if we don't have strong open source models, we're going to be locked into hyperscaler APIs for life
- piddly little home GPUs don't do squat against this
Why are you building for hobby uses?
Build for freedom of the ability to make and scale businesses. To remain competitive. To have options in the future independent of hyperscalers.
We're going to be locked out of the game soon.
Everyone should be panicking about losing the ability to participate.
Play with your RTXes all you like. They might as well be raspberry pis. They're toys.
Our future depends on our ability to run and access large scale, competitive, open weights. Not stuff you run with LM Studio or ComfyUI as a hobby.
I don't agree that we are being left behind with regards to AI, I believe it's simply not worth participating in. I hope it all comes crashing down.
That's not the right perspective to have.
Also, the only thing crashing down will be the economic participation of everyday people if we don't have ownership over the means of creation. Hyperscalers will be just fine.