Comment by apexalpha

I think one could describe LLM optimization as "hard but not a moat". Years ago, optimizing neural nets was described "graduate student descent" - it's tricky but throw enough conventionally smart people at it and it will happen. It's like tuning a hot rod and finding a reproducible bug in a large code base. It's hard and there are tricks but not absolute hurdles, no problems waiting for a conceptual breakthrough (and at today's scales, are there any problems waiting for an Einstein to solve? That's an open (AI) question).

I want that to be true (assuming you mean specifically the second half of the sigmoid) just to give me room to adapt to the changes we've already seen; but I've seen comments saying things are slowing down since around when GPT-4 came out.

What is really going on: all the AI labs are doing panicked model releases (and panicked training of new ones) because Qwen4 is rumored to come out end of October and is rumored to be very nice. Question is: is it another "Deepseek-moment" nice? Or just nice?

Btw: with Qwen4 I mean the next large Qwen model that is based on the Qwen4 architecture (Qwen 3.8 flash next was "almost" based on the new arch but obviously was a small model)

I guess all predictions age like milk, but here's one:

There's a law of diminishing returns at play here, and doubling the energy cost of training to wring 2% more performance out of the technology isn't going to be very useful, because most of the problems it is capable of solving will be solvable with the previous-gen 98%-as-good model.

("there's a law of diminishing returns at play here" is an article of faith. But then, so is the belief that these models will keep getting better).

Surely there is a point where algorithmic improvements will be more cost effective than buying more hardware.

But the old models still exist at trivial marginal cost. The frontier models would need to dominate every price point to really take all and so far they haven't been.

Not just compute but energy. Most of Europe has no access to the cost effective power generation needed

Framing it that way hides all the levers used to tilt the playing field for US companies, no?

The government that removed restrictions on how private companies can access capital after a certain scale (the JOBS Act), that removed the need for private companies to report as if they were a public company after a shareholder threshold was crossed, superpowering the access of wealthy private investors to get in earlier in a growing company while at the same blocking the public from participating in funding growing enterprises at an earlier stage (since it required companies to IPO much earlier to access capital) which allowed retail investors to also reap the rewards on funding them early when they grew to become behemoths (like Amazon, Meta/Facebook, Google, etc.).

It's not fair in either place, the USA has its own model of unfairness, China has a completely different one. The difference is that in the USA the government allows private investors to become more powerful than the State (outside of the monopoly of violence) while plunging the rest of society into increasingly more precarious lives while in China the State is the power and its legitimacy only exists while the population feel they have a better life.

If you think about it, the US labs are heavily subsidised too. Not only they receive billions in state funding, the administration is also prepared to engage in trade wars to help them.

I think the Chinese government is backing their labs by less direct means. For example cheap electricity and investing in chip manufacturers such as Huawei and cxmt.

Deepseek specifically, is known to operate with minimal resources. The entire company has around 160 employees and every model they release must break even within ten months.

Well, they are helping 17% of the world's population, and the US is currently actively engaged in trade wars and economic warfare or explicitly attempting to leverage it's hegemony against its long term alliea for short term gain.

There is as much to criticize about American hyper scalers and AI labs and the lack of interest in helping humanity or contributing to open source, but that might not be as popular an opinions on this site.

> The reason they release the AI models is economic warfare against US

The story is so much more complicated than that, to the point that this economic warfare theory is basically a meme.

Chinese models are open because they don’t have a choice. “When you trail the frontier, openness maximizes reputation per unit of capability. The moment you lead, you close.” [0]

[0] https://earnedintuition.substack.com/p/involution-without-ex...

So is American models. They are subsidised heavily but still can't provide cheaper access. Their fault is to assume all countries can afford them.

> DeepSeek, and other Chinese models, are heavily subsidised by the Chinese government.

DeepSeek is basically a research lab founded by a hedge fund guy, more than anything else.

> The reason they release the AI models is economic warfare against US, not because of charity or kindness

There are many other reasons Chinese companies releasing models open-source or open-weight makes strategic sense.

A really easy-to-understand example is a company who has a near-monopoly on "serving video content" releasing a video model openly.

If you can be relatively certain that video content created by a model (which you have trained, using data from your own platform) will be ultimately served on your own platform, thus generating revenue from watch-hours, it makes sense to make those models as widely-available as possible.

It's also a net-positive if people use your public research to build better video models, because - again - you are reasonably certain that the even-better content those new models produce will be watched on your platform.

The alternative would making models harder to access and learn from (broadly, the current western model). Many would argue that Google, in choosing to not optimise its video generation models for "availability", is directly causing less content to be uploaded to YouTube. This is the trade-off.

I don't know much about DeepSeek's financing specifically, which obviously doesn't release video models - so I don't know how directly this analogy runs, or who directly benefits from the extremely evident rising tide that the public release of DeepSeek's research creates. However, this does not negate the broader rising-tide effect of the scientific method.

It's certainly also true that it's geopolitically beneficial to be able to undercut American labs' models. If I ran a global superpower, I would probably want my country to be technologically competitive too.

But Chinese companies are already serving a huge volume of customers in a complex, existing marketplace, before even thinking about the US market, and it's overly simplistic to assume that their entire strategy revolves around economic warfare directed specifically at the US. It's more nuanced than that.

This is, of course, without even getting into opening the can-of-worms around whether US economic policy also results in the US state functionally subsidising technological innovation, how comparable that is to China's model, etc.

I'm not sure "subsidised" is the right word. If a government funds research and the results are released openly, that's just publicly funded research. It's how a lot of science works in the US and Europe too.

The open-weight models are a great boon to all American companies other than a handfull of Mega Corps in the A.I business. Care to explain how this is "economic warfare against the U.S"

Same goes for the US companies as well. Current US administration wants US to win the AI race at any cost and China is the only competitor left in the race. Winners always write the history or in this case the future of humanity

Economic warfare against the US? I mean maybe against specific US companies and stakeholders but on the whole it seems like it's good for the US economy as well as the rest of the world, kind of like supplying free electricity would be

And OpenAI and Anthropic are just in for the love of the game? Our of sheer desire to help humanity?

It's one of these rare cases where intention is not what is the most important - the net benefit for consumers and companies outside of the USA is indisputable.

Which is the best kind of reason.

It is great for everyone except for a few people who want power over everyone else, and the fact it is not charity of kindness makes it more sustainable, because charity and kindness is quick to go when big money and politics is involved.

I want more warfare like this. Building stuff instead of destroying stuff.

Not sure why so many people will vehemently refuse this idea. I won’t say it’s 100% true but it would be foolish to dismiss it. China is very much an adversary to America and has made it pretty clear they want to be a dominant leader of not the new world leader. Not here to evaluate what is good or bad. Keep in mind historically China has aggressively fostered industry (not unlike the west) but sometimes even more aggressively.

They are also releasing the weights so I am more inclined to think they are better than most.

Well, US models are economic warfare too, of course.

> The reason they release the AI models is economic warfare against US, not because of charity or kindness. It's great for us consumers, but the goal is not to help humanity or open-source.

You put it like US has a goal of help humanity or open-source

What about cheap solar panels and electric cars, is that economic warfare too?

To say that US model providers have a close relationship with their government would be putting it mildly.

That would have been a scathing criticism if anyone believed any of the American companies have the goal of helping humanity or open-source.

Notwithstanding that Chinese publishing methods actually does help both humanity and open-source.

It just so happens that helping open source is the result. Image how far behind we’d (the hackers, not the moneymen) be as a sharing community be without them.

At least DeepSeek didn't build its models on government subsidies. It came out of a quantitative hedge fund, so funding was never really an issue for them.

Whatever their reason, I'm glad they do it. This tech shouldn't be monopolised by a handful of closed, profit-seeking corporations.

So you’re telling me warfare doesn’t having to be blowing up schools and children, skyrocketing prices, crippling sanctions, and all that shit? Can I sign up for more of this warfare.

If your belief is accurate, we should expect China to short the IPOs of Anthropic and OpenAI and release better frontier models immediately after their IPOs.

Does anyone think that likely? I have no clue or bias.

Economic warfare against the US is charity and kindness to a sizable portion of the world's population, especially when the US uses it's global hegemony as warfare against them. Neither system is perfect, but lets not be disingenuous.

Precisely. It's similar to their practice of aluminum dumping to depress US aluminum prices, which causes our aluminum mines and mills to close.

Claude, and other American models, are heavily subsidized by investors (and the United States government).

If the means of achieving their goal are sometimes mistaken for charity and kindness... are they the good guys?

To be completely fair, write this kind of paragraph for every AI category. Would love to see your take.

I like when bad intentions produce good results, I'm tired of seeing the opposite in practice.

At some point though _the purpose of a system is what it does_, regardless of their intent.

How do you know? You’re not speaking for the Chinese government, are you?

> DeepSeek, and other Chinese models, are heavily subsidised by the Chinese government.

Yes, because everything China does is against the US. That's all they think about day and night. God forbid they want to corner the global market or have a genuine business case. How dare they provide options for those who can't afford a measly $200 a month? How can we let Chinese labs publish research for free for the whole world so that they can benefit? The nerve! To think they can use soft power instead of military might! I mean, Anthropic and OpenAI are the last bastions of human kindness and charity. Right?

Right?

I’m neither in the camp of Chinese or the Americans(collective West) in general..

As a neutral party, this characterization is crazy.

As if the AI companies - Claude and OpenAI are guardians of freedom and humanity and very charitable to the global society without any self interests… “Chinese models are subsidized by the Chinese Government, therefore they’re inherently bad for humanity” is a highly propagandist argument. The politics of US vs China may be whatever it is in reality.. You have one company releasing their models for cheap, actually open sourcing their trained weights, and publishing details of their optimizations and learnings for others to use. The other camp actively “aligning” their models, nerfing their capabilities, hyping their swarm activities from poor sandboxes, and trying their best to lock users into their harnesses and walled platforms. They are subsidized by the capitalist VCs who are essentially waiting for their payouts..

At some point, one has to see things for what they are and evaluate their own reasoning..

I’m happy to stay provider agnostic, try all models and cheer any useful progress as open as possible.

> are heavily subsidised by the Chinese government

We hear this about literally every industry the Chinese excel in - that it's only because the government subsidizes them that they succeed. For chip manufacturing, for batteries, for EVs, for solar, for AI. I don't see how the chinese government can afford to subsidize all of these industries and still have them contribute to the GDP.

> DeepSeek, and other Chinese models, are heavily subsidised by the Chinese government.

So were Amazon and Uber by the US, which have now established monopolies across the globe. To the countries suffering from those, there's zero difference with China doing it to solar. Actually there is, at least solar got them cheap renewable energy in return. This would never have happened in the US because big oil interests would make it take decades. That's the reality.

You need to spend 10 years outside the US, deprogram, and then go back.

> DeepSeek, and other Chinese models, are heavily subsidised by the Chinese government.

US companies are burning colossal piles of cash in ways that makes it unclear if it qualifies as dumping, not to mention their deep ties with the country's regime.

Claiming that companies from a country have ties to the regime and burn through cash is a very miopic accusation.

I wish US was rich enough to subsidise development of open source science and useful open AI models. I wouldn't mind US waging this kind of wconomic warefare against China or everyone else on the world.

Charity and kindness is not a motivation, it's an outcome of what you do.

Meanwhile, American AI models are heavily subsidized by stock market speculation. Ultimately, the subsidies from both countries are flowing out of the pockets of individuals.

US is basically doing economic warfare and bullying against everyone else ATM :)

What if economic warfare against the US does help humanity?

Why are we assuming a strong US is necessarily good? As a European, I have seen plenty of evidence against that stance lately.

I understand that Americans might prefer a strong US. But conflating them with humanity is a leap that I don't think one can make without any backing.

A conspiracy to make the US look bad by being better at producing all the goods and services the world needs at a reasonable price. Have they no shame?

What's wrong with governments subsidizing scientific work?

You think extremely US-centric. China has a different economic model than US and your rules for a specific kind of Capitalism may not apply to them. You assume a country of 1.4 Billion people is obsessed with a couple of foreign AI companies. What if they don't care.

> but the goal is not to help humanity or open-source

Ok, but so what? That's what happening so far. Even a repressive totalitarian government I wouldn't wish on my worst enemy does some good sometimes.

If the Chinese cheap/open models rise up and destroy us, thats on us for giving them access to the tools to do so.

Not really this is an X algo conspiracy. Up until recently the Chinese government wasn't even that invested in these companies. We're talking very very small grants compared to training costs.

Its very xenophobic of you to say China has zero intention of helping humanity, and just wants to "wage economic warfare".

Last time I checked, it was ourselves (USA) waging economic warfare on 2/3rds of the world.

I dont get this cope people have where people have this idea that its impossible for a Chinese company (that make billions of dollars) to have done something by their own merit, but instead its always some Chinese Communist Party conspiracy where the main goal is to destroy America.

Lay off twitter for a bit.

[dead]

[flagged]

[flagged]

Why the anti China propaganda from you? The United States is doing the same thing here and making American multi-billionaires even wealthier. The greed of American AI, GPU and memory and storage corporations is a black hole on availability to humans around the World. The result is a massive financial bubble promoted by the American Government to the detriment of our citizens. The USA has an AI ponzy scheme shuffling the same money between data center owners (Oracle & X), Nvidia and memory & storage vendors.

Deindustrializing a country is not something consumers can achieve. It starts at the top level, with politicians who construct a financial system where it's more profitable to speculate than to build or invest in real businesses.

Except they are open with the tech which is easy to replicate. All new models lowering cache prices is the result of DeepSeek's publications.

US economy is driven on quarter to quarter short-sightedness of stock gamblers and CEOs that have to secure their bonuses and perks.

Its them who chose to outsource heavily, not US consumers.

There are no victims here however, both benefited from the arrangement. The consumers and capitalists.

> The reason we can't manufacture a grill brush for a reasonable price is the result of years of Americans choosing the cheapest price.

No, it's the result of US leadership letting this happen. This is clear since China themselves would not let this happen. US leadership did nothing because they were best friends with the people who did it, and did not care one bit about their population.

Evidently not given that this model is on the open source frontier.