Comment by alganet
2 months ago
The part on training is misleading and full of shit.
Training is not a "one-time cost". There is an implied never-ending need for training. LLMs are useless (for one of their main purposes) if the models get stale.
I can use Musk's own argument on this one. Each model is a plane, fully built, that LLM researchers made into a disposable asset destined to be replaced by a newly built plane on the next training. Just incredibly stupid and inneficient.
I know what you're thinking right now: fine-tuning, etc. That is the "reusable" analogy to that, is it not? But fine-tuning is far, far from reusability (the major players don't even care about it that much). It's not even on the "hopper" stage.
_Stop training new shit, and the argument becomes valid. How about that?_
---
I am sure the more radical environmentalists know that LLMs can be eco-friendly. The point is: they don't believe it will go that way, so they fight it. I can't blame them, this has happened before.
_This monster was made by environment promises that were not met_. If they're not met again, the monster will grow and there's nothing anyone can do about it. I've been more moderate than this article in several occations and still got attacked by it. If not LLMs, it will target something else. Again, can't blame them.
If we're trying to figure out a reasonable number for how much energy a single ChatGPT search uses, it seems weird to factor in all future training of all future models. It would be like trying to figure out the carbon cost of a single plane ride and trying to figure out what fraction of additional plane flights that one ticket incentivizes. It's too murky to put a clear number on and probably doesn't add much to the cost on its own. I tried to make it clear that the post is about your own personal use of ChatGPT, not the entire AI industry as a whole.
I agree, it's an impossible task to measure it. Almost like a labyrinth you can get lost if you worry too much about it.
The environment is not a personal issue. You can't solve it just for you. The whole idea of making it personal is so that a collective aspect of it would flourish. What are you trying to flourish in people's minds?
I added in the article that I think this is all the wrong way to think about environmentalism anyway and that people should spend most of their time working on systematically changing the energy grid to renewables. But I keep meeting people who are worried about their personal emissions from ChatGPT, so I wanted to make it clear that if you are worried about that ChatGPT isn't really a drop in the bucket compared to other things you could change.
I am also wondering, why they don't do online learning with their LLMs. Seems silly not to. Maybe they are performing some irreversible post-training steps, that do not allow for picking up training again afterwards. In any case, being able to feed an in production model new training data would be one of the simplest ideas to come up with.
> LLMs are useless (for one of their main purposes) if the models get stale.
I don't think that's entirely accurate. A lot of people deliberately continue to chose to use the older GPT-4 despite it not being updated since June 2023.
GPT-4o has had releases in May, August and November of 2024 - so about one every 3-4 months.
Anthropic's Claude 3.5 Sonnet was released in June and had a single update in October.
Personally I'd rather have a model with excellent summarization / tool using abilities that can look up recent facts about the world.
LLMs that can replace search (their primary and self-declared goal) cannot survive without repeated training. As I said, this is one of its main purposes.
The other main purpose (military application, surveillance, autonomous psyops) is also highly dependent on continous training. Without it, properly educated healthy humans can overcome its reasoning power very quickly.
All other user profiles are just cannon fodder. Companies don't give a fuck about people running older models. They'll do whatever they can to make you use a more recent one.
That's why I'm being provocative with the "let's stop training new shit" argument. I'm aiming for the heel.
If somebody told you that LLMs are a good replacement for search that person was misleading you.
1 reply →