Comment by petesergeant
17 hours ago
> That is the danger of open source models. They destroy monopolies and shrink margins.
It's unclear that the leading open-source models could exist without having distilled the frontier models. If that's true, the "danger" is that these upstream models don't get built any more (because of the tremendous cost), and we end up just optimizing what we have. Also unclear whether this would be a net win or loss for humanity.
The tech is at the point where synthetic data is almost high enough quality to train further models on, AND there are so many human users of cloud AI, that their usage can be gathered and trained upon (it's human generated and used in real world situations, so, it is of a good enough quality). The cat is fully out of the bag.
This deals with half of one of the three scaling factors only. Presumably data production (choosing your training data, etc), model size, and training time/power still contribute huge costs. Who's investing that cash if open-source models will distill out enough to compete without the investment costs?
There is a community of people who compile LoRAs of their favorite subjects/styles in order to fine tune models. There is currently a bit of a gap between LOrA creation an integration of that data into larger models but I can see serious hobbyists stepping up to that plate soon.
The upstream models weren't built by god, they were built by distilling the world's knowledge and cultural products.
If OpenAI/Anthropic stop doing it there's no reason to believe someone else won't. Or that we won't come up with a different approach.
> they were built by distilling the world's knowledge and cultural products
... and a bunch of RLHF and proprietary training data etc etc, but regardless, this is still an exceptionally expensive process, that someone somewhere is paying for.
I'm OK with this.
Anthropic is zooming to $100 billion in annualized revenue.
The tremendous cost is matched by the tremendous business output.
In the span of less than five years, starting from their first $1 of revenue, Anthropic + OpenAI will have produced businesses doing at least $150 billion in combined sales. The demand isn't reducing at present, it's climbing, they can't come close to servicing the total demand for their frontier models.
SAP is under $40 billion in annualized revenue. They're 54 years old.
If SAP sold $1 bills for $0.50, they too could rapidly outpace Anthro/OAI's growth. And when sane minds ask about profits or a defensive moat, the response is "but look at our revenue bro".
The same exact thing was endlessly claimed about Uber. They were guaranteed to die. They were selling rides for less than they cost. Uber was a commodity app anybody could MVP in a weekend.
Market captured. Whoops.
Uber now: $55b in sales, $6.7b in op income. They won.
And to be clear, OpenAI can easily permanently subsidize a fake low cost for their LLMs if they get off their asses and attach a monster ad network to GPT already. They're years behind where they should have already been with ads, the easy money made it so they didn't have to think about it. If they miscalculate the timing on switching over from easy money (market) to ads (also easy money) it could certainly kill them.
OpenAI can knock out a lot of competition by building the largest ad network, then undercutting the cost that consumers see. The margins in ads make that possible. Google already knows how that works, they're doing the same thing right now.
I'd rather have the OpenAI + consumer base + ads position, than being Anthropic. China is a far, far larger threat to Anthropic over time. OpenAI is building a service business more comparable to what Google does. Chinese models don't actually threaten either OpenAI or Google insofar as their positions are user capture based, whereas constantly improving Chinese models are a never-ending threat to Anthropic.
2 replies →
According to who? There is no agreed upon anyone that says that?