Comment by AaronAPU
2 hours ago
They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
In some cases, such as space exploration I don't really care who does it but that it happens - sure, would be nice if my favorite power block did it but I will still celebrate it when someone else achives it.
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.
All the worst outcomes involve taking this technology away from the public.
The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
2 replies →
Anything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.
2 replies →
astra is more parlor tricks than real gains tbh
i swear they trained in on threejs in particular so those idiots on twitter could spam their garbage demos
1 reply →
They've not plataued but they're certainly not as impressive as the hype would have them to be.
The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
2 replies →
Impressed with the model performance or the chatbot/agent performance?
Really? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value.
For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine.
Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious
9 replies →
The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
> Quality data can be created synthetically at an exponential rate as models improve
No it can't? Every time the labs try this we see model collapse, e.g. shoving goblins into every conversation.
And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
5 replies →
Can you provide sources for these claims?
1 reply →
> It explains why the “we need to race China” concern suddenly vanished in the discussion
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
Seems obvious to me, though, that you can't beat China by pausing when they don't, and China is not likely to coordinate.
China might pretend to coordinate in public but you can assume their secret research labs are racing full speed ahead.
6 replies →
idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
I remain amazed that the idea of the USA being a coherent, unified rational actor one can describe as a "Nation State" has survived the current administration.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
Yep, the cost of DDR5 skyrocketed to keep Mr & Mrs open source from parallel developing their own at home solution. Because Governments can't be priced out of the market. China can't be priced out. VC's want open source locked out of the running if possible. I also don't doubt that the models that are released publicly are somewhat handicapped versions of whatever the government get access to.
This is like a bizaro second amendment debate for the 21st century.