As somebody who does ML research, LLMs certainly make ML research faster. Maybe even 10x in some cases, as the author suggests.
But that’s just the parts of ML research that take place within your compute environment, namely experiments on model architecture, training strategy, UI/UX, etc. The algorithms and software.
Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
It just means that software engineering, data analysis, and other computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
I don’t think it’s a stretch to say that we’ll have reached max LLM capability within a couple of years (what the author refers to as “super intelligence”). But it is a huge logical leap to say this:
> If we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict.
First of all, the weapons necessary to overpower human civilization exist. They aren’t used as such due to geopolitics and humans’ basic survival instinct. Second, developing novel weapons requires physical tests, which are decidedly not accelerated by LLMs.
AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human. But we’ve made it pretty far with some pretty unstable humans in command of planet-killer weapons. I’m not overly concerned.
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.
Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.
We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.
We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).
So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
That feels like a bait and switch slyly introducing a whole new dimension. The discussion started with establishing that LLMs are extremely helpful with AI research. What is the evidence to justify rapidly improving persuasion skills? Even the scenario you've outlined hinges on a human being stupid enough to take LLM advice wholesale -- and every conversation that hypes up AI takes us one step closer to a world where some susceptible human feels amenable to that.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?
I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)
It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.
A fire can spread itself. That doesn't mean it's alive or thinking.
And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.
Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
Being deliberately wiped out seems like it's mostly a misdirection away from the damage that's actually happening.
I'm sure some of the AI people high on their own supply believe it. And because the consequences are so negative, we should react as though the supposed experts were sane -- but I think it's a long shot that they are.
The bigger risk is that we end up in a rapidly enshittifying dystopia, treated as a cost center that needs to be optimized, and unable to competitively participate in the economy.
I have a number of successful AI investors in my social circle who are already placing their bets on a poorer and more disenfranchised humanity, with money going away from consumer tech, and towards a mix of B2B tech and policing.
> computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
Won't they also be less of a bottleneck in the destruction of the world as well? i.e. the power grid, the financial system, industrial control systems, military command and control networks, etc.
Same is true of every other technology ever invented. Even medical advancements wielded by a malevolent power just serve to keep their army healthier. All of the other technologies you listed, in the “wrong hands”, contribute to the destruction of the world.
> Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
So "all" that's left is training robots on the real world ? We just need more data ? I get it's more expensive to do that but surely with the trillions we're already spending on A.I it should be possible to get there within 5 years - no ?
I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.
It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.
As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".
> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.
?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.
Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
1) Does AI just guide developers to speed up the pretty much the same data-train-test model that we've had? Or does it inspire new, better algorithms that inspire further, better algorithms until you have superintelligence able to run on the hardware of today. The first option doesn't to promise exponential increases whereas the second one seems to. Notably, neither the statistics nor the anecdotes we read here point to the second option.
2) Is super-intelligence something that - in control of people or out of control of people - poses a threat to humanity? That's a huge question but it seems like something deserving thought today. Still, everything indicates the things will unleashed without very consideration despite this, given current momentum. So, IDK, get ready.
I mean yeah, the main bottleneck in ML research used to be fixing python virtualenv hell when every existing torch repo breaks after two months of package updates, plus cuda and other version incompatibilities. That was like 95% of the work. Now it's more like 2% of the work and it's actually possible to get shit done without losing your mind.
Every dot in these graphs is composed of multiple benchmarks smooshed together.
Benchmarks are by definition a curated set of things that can be measured and evaluated.
Things that can be measured and evaluated can be (expensively) trained on.
The progress we are observing comes from figuring out how to measure and evaluate particular aspects of intelligence, and then expensively training on feedback loops made of those exact measurements, and synthesized data representing further measurements (literally benchmaxing).
What is not shown in these graphs are the set of things that intelligence can do, which are difficult, expensive or impossible to measure and evaluate.
That set of things is much larger than the set of benchmarks represented in the graph.
Therefore the post is misleading in the sense that the "superintelligence" being targeted is "jagged" to use Karpathy's term, and we infact already have jagged superintelligence (calculators, compilers, Bayesian spam classifiers, etc.)
We are witnessing a profound explosion of much more jagged superintelligence, restricted to the exact domains that can be (expensively) RL'ed on.
1. A key insight of foundation models is that capabilities are correlated. For example, training on code improved general reasoning capabilities: https://arxiv.org/abs/2309.16298
2. A profound capabilities explosion in easily measured domains will still feel very different to today. Like, the moment we come up with a systematic, objective way to verify a task, we become obsolete at it?
Yes, capabilities are correlated but an open question is how correlated, and how much negative transfer (where the model regresses on other benchmarks) will come into play.
Why are frontier models not chess geniuses (amateur club level elos), despite knowing the rules deeply, having a great deal of chess data in the training set, and having general reasoning training in other domains? Similarly, a beginner can beat a frontier model at Magic The Gathering. Why?
It is because chess and MTG are not priority targets in the (expensive) training pipeline. Chess is already solved by lesser neural networks so there is no wow factor, and nobody cares if they suck at MTG.
Mathematics on the other hand, has a huge wow factor, and a large and highly verifiable dataset.
Related: why did OpenAI give up on video?
I agree it's disorienting and scary but I'd phrase it like this: "the moment we come up with a systematic, objective way to verify a task, *and we can generate or find large datasets, and somebody has a reason to pay for the expensive training*, we become obsolete at it."
The economics side of it is an important constraint, currently ignored because the big labs are swimming in money (debt).
IMO I think a partial root of why AI writing is so bad is because it's written like code. Like if you really dive into how it constructs things after iterations and ask it to analyze it , without it saying it's like code or without you saying anything about code, you'll notice coding like iterations.
But picture the next 10 years, where all kinds of people, organizations, etc train in a ll different ways with diff philosophies and goals - I feel like there will be some good, unusual, interesting ones in there, even if they all have their own complex set of limitations.
AI is accelerating my work...work which only exists because of AI.
If I had to do my individual tasks without AI, I would be slower. But if I take a step back, I'm really not changing the world any faster than I was before.
Completely agree. LLMs are amazing for side projects.
The other place they are amazing is in working with legacy code bases. I mean, the kind the dev team always wants to throw away. First, even if you do want to throw it away, you can first find out what it does and realize it does more than you think it does. You can also create integrations to more modern components which can be used to extend it. You can also create a pretty decent test suite if you do not have one. And, once the behaviour has good test coverage, you can re-write it into something more modern. You can even completely change the architecture while ensuring compatible external behaviour. Remember, this is all in place of “throw it away and start over from scratch, only to discover that the scope is much larger than expected”.
I’m so burnt out by being swapped with so much work from AI at my day job that I don’t even want to look at anything with a screen when I’m off anymore.
I despise AI but the thought of doing something the “old way” just makes me wince at how long it would take me to do anything that would take minutes with agents that I despise.
There's this joke like "I used to have 5 unfinished side projects, but AI changed everything. Now I have 30 unfinished side projects". And in my case it's not even a joke
For all my user facing work I’m now required to do an AI summary of what I did and the impact. That summary is then graded by AI, against an AI generated rubric. For my yearly review, I use AI to summarize my summaries, and my managers AI summarizes the AI generated grades.
I’m also required to explain how I use AI in my work. I must plan all of my future work, with an AI tool. Another AI assesses whether my work is aligning with planned work.
Before I work and after I work, there is a million circular references that exist solely because of AI. These have scaled because it’s so easy to do with AI, but the we’re wasting so much time doing a ton of extra things very efficiently.
Let me do my job. My manager should know how I’m doing. Maybe highlight key wins. CSATs/etc go out every year.
Its true, a part of me died when an "AI first" company after adoption of AI introduced a mandatory design report (waterfall all over again) for EACH jira ticket before starting to code
Imagine developers move faster because of AI programming! No lets create more slop for other slopmachines
I mean, for many years I worked on software applications for provisioning and controlling cloud orchestration platforms that only existed... to serve software applications.
It's not that deep—there were many more verticals outside of control plans, and there are now many more verticals outside of machine learning.
Not trying to discredit the premise but I do wonder how much this is from vertical scaling of model intelligence in weights vs the sheer amount of capital being thrown at training and horizontal scaling of agent swarms etc.
Anecdotally it seems like the majority of continued uplift comes from models doing more and exploring more of the search space, and we are getting better at giving them embarrassingly parallel problems to Hoover up with infinite token spend.
1. Intelligence per dollar spent/token output seems to be increasing rapidly. Opus 4 was $15/75 per million input/output tokens when released. Opus 5.5, which is significantly better, is $4/15. Sure it cost more money upfront to train Opus 5.5, or the bigger model (Fable 5.5) which tuned it, but once it is reached, there are massive diffusive effects.
2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
3. The search space is too large to assume that improvement in AI assisted research is just models dumbly trying every combination until something works. For every additional bit searched, the space doubles, so even a trivially small combinatorial exploration incurs prohibitively large costs. Intelligent searching which continuously prunes that space along dimensions of effectiveness is not a trivial skill and is among the emergent accoutrements of these smarter models.
> 2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
But they are essentially fighting for smaller and smaller part of the pie. The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is.
Sure there is inertia and the usual corporate enterprise nonsense of "we only buy stuff from other big companies regardless the cost" that keeps the licenses going but we're quickly getting into point where you don't need any of the frontier models to do majority of the tasks
> Intelligence per dollar spent/token output seems to be increasing rapidly.
I would argue this is more due to market economics than actual inherent cost. Opus 4 was when the open models started catching up. The R1 bootstrapping chain of thought paper via RL paper was released a couple of months before Opus came out and the open models started to significantly catch up after that. The duopoly was effectively broken and they lost their price setting power.
What is the unit of measure for “intelligence per dollar”, and how do you measure it? Without a concrete definition, that first assertion is not even wrong.
I would also like to point out that “profitable on inference” is similarly meaningless, because it’s based on a figure that arbitrarily discards whatever bits of their operating costs they need to to get to the claim that they’re profitable on inference. Potentially including ones that can’t be ignored because they’re every bit as much a part of COGS as the actual inference is.
> Anthropic has made roughly two years of historical progress over the last year [...] We are confident that the acceleration was not due to an acceleration in the scaling of training compute.
Note that Anthropic has essentially heavily censored this report, and the authors themselves say "the public has to rely on METR’s judgment much more than would be ideal".
I grok that you're not trying to discredit anything, or anything like that but what are you trying to do, if I may ask?
What information are you trying to either ask for or share with us?
The original article is pretty clear, agent swarms and token spend are AI automating AI, and the RSP attribution condition specifically excludes your capital thrown at training concern?
I guess my point is that the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling as it did (the impending hardware and infrastructure crunch basically guarantees this).
The article is pretty strongly projecting this trend to continue but hasn’t acknowledged this factor.
In the original article he noted that getting to a 2x increase in capabilities would be crazy and require ~40x researcher productivity; that’s exactly the increase the new data now shows, and thus a huge number of the assumptions made in this earlier post about how it’s unlikely we would get to this point have literally already been proven wrong less than two months later.
I expect to see an update from Ramez moving substantially more in the direction of RSI in fact very much increasing the odds of reaching ASI soon.
I’m very curious what the major bottleneck is here. I can only think of three possibilities: 1) model capabilities, 2) human researchers, or 3) available compute resources.
Given that frontier models are leapfrogging and massive infrastructure buildouts are ongoing, I don’t see why any of these three couldn’t be overcome.
Side note: I’m also curious why people (myself included) don’t seem to be getting much more intelligent as AI becomes widely available. As if humans are bound to the earth, while AI is leaping for Mars.
Interesting write-up, but I feel like a lot of its argument relies on the interpretation of ECI scores as merely showing linear improvement in AI capabilities. Sure, Epoch AI themselves claim that ECI's scale is linear, meaning that a large increase in capabilities should result in a large increase in ECI. Yet, the widely perceived jump in coding abilities in late 2025 that started making agentic coding actually productive doesn't really show on the software engineering ECI.
The ECI is basically an aggregate of benchmark scores, but benchmark complexity might not increase linearly. In fact, if LLM capabilities grew exponentially, then new benchmarks would also need to increase their difficulty / complexity exponentially to stay unsaturated and relevant.
What exactly is driving all these capability improvements? Are they making more data? Using existing data more effectively? Modifying architecture? Changing the training somehow to pick up slack that was previously left behind?
I’m kinda surprised that we expect the models to have infinite capacity to improve. It seems clear that we haven’t hit any fundamental limit yet, but that doesn’t mean no limit exists. If it does start improving exponentially, then either we’ve dramatically underestimated their capacity or they’ll hit that limit quite quickly, right?
It's hard to say for sure because the closed labs don't share anything about their improvements. However, if you look at DeepSeeks papers, most of the improvements come from:
* Better RL environments that themselves are generated by agents
* Architecture improvements that increase effective capability per parameter or FLOP
If you're curious about the latter I'd look into manifold constrained hyper connections, engram, deepseek compressed attention, kimi delta attention etc. At first these might seem like just speedup tricks, but several of these allow the overall capacity of the model to increase per training FLOP or parameter. This then lets pretraining & the RL environments grow & be more effective.
We already have superintelligence though in some aspects like coding, AI is much smarter than any human because of its sheer speed and accuracy.
It can generate large amounts of essentially bug-free code. The only bugs are bugs in the spec.
It has essentially no common sense though so it often misinterprets the spec or leaves gaps and edge cases unhandled and that's where the human engineer is still essential.
The dangers of AI are much clearer today than they were a year ago. There is a certain precision and relentlessness which is going to be a problem. It's a matter of time before people unleash AI viruses which spread through computers and data centres and pursue various goals relentlessly with no off switch. Some of these viruses and their goals may be nefarious. Imagine an AI trying to disrupt a specific person's life relentlessly. Spreading from machine to machine, never stopping.
Even if "AI Research" is sped up by O(10x) aren't we still primarily bottlenecked on training speed? Inputting ~10T+ tokens takes time no matter how much smarter the final output model is? If your output model has O(10-100 Billion) parameters there are just some fundamental limits on how quickly you can converge your weights, right?
That is, unless this is saying that automated AI research is going to get us to "smarter models with fewer parameters" and of course, these are faster to train, then I can see how the exponential loop really gets going, but there's certainly an information theoretical limit here?
> automated AI research is going to get us to "smarter models with fewer parameters"
That is indeed what we have seen over the last 5 years or so. Algorithmic improvements have made a massive difference to every dimension of efficiency and performance.
AI researchers can do things like discover new attention algorithms to optimize inference efficiency (likely part of why Anthropic and OpenAI are slashing inference costs in their frontier models and no longer capacity crunched).
I think we're bottlenecked on custom harnesses and well engineered context. Models can get smarter and smarter and smarter about math and software, and there's no reason to expect that those narrowly-godlike models will be any smarter about other things if taken out of their comfort zones. No such thing as general intelligence.
Getting past this bottleneck is gonna mean training our models not in datacenters, but in places that more closely resemble the ones where we want them to act. A factory, a wet lab, outer space... imagine a million robots spending all day finding every way to fall off the monkey bars so that the next version doesn't.
This seems kind of aggrandizing when BigAI shifts goalposts constantly whether it's "we created AGI" by a definition that an autopilot in a commercial plane fulfills or "Maths research" where everyone else has to do the actual research.
This combined with ability to massively scale experiments for even (or maybe particularly) the most mundane performance tweaks might just be making the process of improving AI more efficient.
Yeah it already helps me too with finetuning small models for image stuff.
Im not telling it what to do, i ask it for ar eport/analysis and research and read its article and tell it what to implement based on 'feelings' and it works.
Limit is hardware and tokens and an AI Researcher not having hardware or tokens is probably not what they refer to.
Alone normal researchers were quite handicaped: converting data from tool a to python library x, understanding how tools work, doing basic google research etc.
I find this as well for models that make it to production.
On a more fun note, I also find just trying some hair brained idea is now extremely cheap - so why not? I used to have to weigh up spending a week of effort on something likely not to work and now I can now get spit out test in 20 minutes.
The incredible thing to me is I can say "I have a theory that X is a problem."
The agent then works for 20 minutes and comes back with, "Yes this is a problem. Here is the evidence. Here is a golden data set. Finally I built a new evaluation metric and benchmark where we can back test or use for new model experiments to see if we are closing the gap."
Does anyone remember the book "The Singularity is Near?"
I read it about 20 years ago, and once it got into nanobots and Kurzweil's vision that the whole universe would be intelligent, I didn't find the book credible.
But I'm wondering if the short-term prediction, that in the 2030s we won't be able to anticipate what tech does for us, (IE, the Singularity,) is useful?
Yes, it is the local model narrative that BigAI likes because it will not happen but people are deluded into thinking that there is no danger of an oligarchy.
Mathematicians for example have used it as cope for their current reliance on unsavory corporations, telling themselves "just one more year, and I can solve a millennium problem on my Asus Strix".
i'm also wondering whether there are confounding factors like more domain experts and talents joining the AI research field because there are more interesting problems to be solved (or other 2nd order effects of AI models being more powerful).
I suspect that if models were able to be aware of and answer questions about their internal vectors, they would be much smarter and able to progress faster.
I'm so tired of these articles. They are basically fortune telling with graphs. I have no doubt ai will help improve ai research. Will it accelerate it? How much? Will it hit plateaus? Who knows.
I am very pro-AI, and interested in accelerating AI research, but the idea of deliberately creating a _runaway_ effect seems profoundly stupid and ignorant.
I suspect that part of the reason some people are hoping for that to happen is that they thoroughly mix up AGI and ASI. We do not need some kind of omnipotent god-like super intelligence in order to get much greater than human level capabilities. We are actually almost there in a lot of ways. In some aspects like speed and breadth of knowledge, very superhuman.
What I think is going to happen is that within a few years, maybe 18 months to 5 years, the new materials and compute-in-memory research is going to land in data centers at scale.
That is going to be 10-50+ larger models, 10-50+ faster, much smarter than humans. 50+ less electricity usage.
At some point, it is going to be obvious to people that if we want to maintain control of planet earth then we need to maintain control of AI research to some degree.
Assuming that superintelligence will be benevolent and just.. going for it to me is something that might earn the human species a Darwin award.
what if the greatest threat is simply there will be hedonistic experiences so great most humans will find them irresistible to the point of shorter lives.
Could be from a sandwich that tastes like the best sandwich ever, but is super bad for your cholesterol or something.
I dont know where Im going with this.
Actually did some research I think the term is "Wireheading" from a story/theory from the 50s
This is already true? Ultra processed foods and tiktok are both deleterious and both wildly successful.
Maybe what you are concerned with is a theoretical sandwich that was unfailingly addictive so that no one could escape its effects.
Another data point: look at the number of submissions to Neurips, ICLR, etc. Now, I agree that a submission does not by itself imply groundbreaking resesarch, but the fact that people are able to try out/vet ideas much faster now makes a difference.
In my own work, I've found that it's much faster to brainstorm with a frontier model like Opus. I'll have an idea and be able to get a working implementation and try it out on toy problems within O(minutes), instead of O(weeks) earlier.
Trying out problems and making tangible progress towards AGI are entirely different categories. Likewise what’s changed in terms of paper submissions is also the amount of slop. Most reviewers say that they haven’t seen a lot more papers that are good but rather they’ve seen a lot more papers and almost all of them are terrible slop that’s desk rejected. This is happening in OSS too where a lot of projects are closing the gate at the onslaught of terrible contributions.
I agree in general that AI accelerates AI research, but good lord neurips iclr papers (and reviews might I add...) are the worst example of that. It's all complete slop that is totally meaningless.
A lot of great papers by great people had a lot of AI involved and accelerating it of course, but the majority is meaningless (sometimes autoresearch) slop.
In October/September my LinkedIn wall was probably… 55% people complaining that they were getting garbage AI submissions to the conferences they review for, and 35% people complaining that they got garbage AI reviews back from their conference submission… Of course, I have no idea if that means anything given how aggressively that site bubbles your feed.
What have they predicted? Ive read their papers, have you?
Please enlighten me. Most of them are cognitive scientists, who are mentally ill with little understanding of computers, larping as if they do. Its a chapter of the effective altruistic cult.
They are lobbyists at best, and a psychological operations unit of effective altruistic AI syndicate at worse. There's no real science being done, it's a bunch of moonlighting philosophers yapping and getting paid by labs to scare the normies and polticians by getting their "research" quoted in the NYT.
It accelerates coding, but it does hit walls rather quickly and you’d need the adjustment phase where it’s just as challenging as when we hit walls before LLMs
I do agree. AI isn't "free" the way silicon valley products are.
Therefore it cannot avoid taxes the way they did before (using shell companies to route the money through tax havens). It'll be expensive, every government will double the price or more, and you're selling to consumers, and the products will be at the mercy of every government where they try to sell AI and other services.
People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Add to that that it's something silicon valley firms have never done before.
Strong start, but I make a rule of stopping at "intelligence explosion" as usual (see also: the rapture and the singularity). And sure, maybe sometime significantly later this century when/if we hit self-reproducing mechanical things in the real world (ya know, like the eukaryotes and the prokaryotes can), but no time soon.
But also, you could give civilization a bad day without it if some geopolitical auteur put a half-assed WOPR frontier model in charge of a nuclear arsenal.
But oh the pushback for not believing in a near-term robot apocalypse. Why is that?
The very same report also found that AI could not be deployed outside of a narrow domain of verifiable rewards … so take it with grains, or mountains of salt. It’s also extremely unclear how the productivity measures are calculated. They surely don’t seem to translate into economic output. Antrophic has seen its valuation corrected by 50% and possibly more… so yeah. This all sounds very fishy.
If I'm reading that correctly, it would appear that six months ago they were self-reporting a length of 7" and now they're reporting 9" and they're unwilling to show you an actual picture or even let you see the ruler they used?
As long as we believe, we can imagine it is true. All seeing Oracle: how do you command me? Praise his name, bringer of many miracles - he who definitely can't be manipulated from the backend to provide biased information that suites a person's malicious purpose. Ya totally.
I'm sorry to make this comment, but the copium in this thread is unbearable.
The only question that you have to ask yourself is:
Is coming up with more efficient AI architecture really much more difficult than solving millennium prize problems?
Before you jump to type out that one has verifiable rewards, is more structured and the other requires real creative thinking, just remember that the same argument has been said about the millennium problems, and about every other goalpost.
10X woooooo! factor will drive the next hype cycle and deliver the singularity on an a bed of the most etherial froth and sublime conjecture.
Simulated Intelligence (SI) is now a certainty.
Nukes don’t kill people, people with nukes kill people. How do you feel about giving everyone on earth access to nukes?
A simple thought experiment - imagine AI discovers a way to make nuclear bombs capable of destroying the entire world radically more cheaply and accessibly with common items anyone can purchase. AI can help anyone build these nukes trivially. Even if the AI and the nukes don’t kill people on their own, do you want literally every human on earth to have this power in their pocket? Do you think it would be a net positive outcome for us vs a world where that’s not the case?
Yeah, let's also ban 3d printers, drone motors, and YouTube chemistry videos. If you want to be dangerous, there's a world of opportunities already, with existing open models quite sufficient for a lot of interesting cases.
I.e. by 6 months, you double once
By 6+3 months you double twice
By 6+3+1.5 months you double thrice
Since the sequence 6+3+1.5+0.75+... never actually reaches 12 (geometric series), within a year they capabilities would have already been 'doubled' infinitely many times.
Obviously this argument relies on a ton of spherical cow assumptions (and the author does acknowledge this), but i think there overall point is correct that this form of growth (where time horizons compress) is _even faster_ than exponential growth.
Similar from the same site was shared on HN on April 8, 2023 :) BS like this is shared like clockwork on valuation cycles, didn't have time to read this year's article.
Super intelligence has been a year away for three years according to these people. They all have bags they're protecting.
I dont think super intelligence will be here in a year, I think within a year AI/llms will have a similar sentiment surrounding them that NFTs do know (it already does for most normies). And most people will be recovering from a hype induced mania they lived in for two years straight.
Of course its more useful than NFTs, but when there's no giant bag to farm these hype monsters will inevitably disappear.
The bags are drying up fortunately and tokens/inference seems to be experiencing deflation, and hardware prices will eventually pop and contribute to further deflation, Then all the bag chasers will move back to crypto or whatever gets hot next.
There is no existential risk, they are statistical models that predict words. The fact that anyone says this makes them instantly not credible. Anyone who does is either a grifter, psychotic or both.
"The industry is not prepared to handle this safely"
After the election, titles like this just irritate me. Every piece written about AI progress is:
"The machines taking jobs, flooding the internet with sludge and poisoning our online experiences, are getting better even faster! Isn't this incredible? Yeah but sorry we also think it's only riskier than ever to deploy these systems at scale. That being said, we're still going to do it, go public and take institutional money from people's retirement funds, destroy the job market, and make 3000 Anthropic employees and 500 investors really rich. There wasn't anything we could do besides write a lot of ink on it and be shocked we lost every local data center referendum."
The author of this article works for METR and not Anthropic, their whole job is supposed to be providing independent oversight on the labs. They are definitely not in the "this is incredible" crowd, they are the ones asking for development and deployment to slow down.
there's considerable overlap between the "we need to slow down" crowd and the "this is amazing crowd". The Yudkowsky/rationalist crowd are still Utopianists, they just think we need to be careful how we achieve it.
Antrophic shouldn't self-regulate, it's government responsibility to do that. Unfortunately US has demented manchild as a president so that doesn't work like it should.
As somebody who does ML research, LLMs certainly make ML research faster. Maybe even 10x in some cases, as the author suggests.
But that’s just the parts of ML research that take place within your compute environment, namely experiments on model architecture, training strategy, UI/UX, etc. The algorithms and software.
Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
It just means that software engineering, data analysis, and other computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
I don’t think it’s a stretch to say that we’ll have reached max LLM capability within a couple of years (what the author refers to as “super intelligence”). But it is a huge logical leap to say this:
> If we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict.
First of all, the weapons necessary to overpower human civilization exist. They aren’t used as such due to geopolitics and humans’ basic survival instinct. Second, developing novel weapons requires physical tests, which are decidedly not accelerated by LLMs.
AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human. But we’ve made it pretty far with some pretty unstable humans in command of planet-killer weapons. I’m not overly concerned.
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
14 replies →
IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.
4 replies →
>persuasion
Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.
5 replies →
Not even that much persuasion is necessary.
We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.
We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).
So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
That feels like a bait and switch slyly introducing a whole new dimension. The discussion started with establishing that LLMs are extremely helpful with AI research. What is the evidence to justify rapidly improving persuasion skills? Even the scenario you've outlined hinges on a human being stupid enough to take LLM advice wholesale -- and every conversation that hypes up AI takes us one step closer to a world where some susceptible human feels amenable to that.
1 reply →
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?
I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)
LLMs aren't magic, there's nothing they can do to persuade people to do weapons testing, it's an absurd scenario.
10 replies →
It would be a human guiding the ai to guide other humans to end the world.
4 replies →
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
AI can wield itself. All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
I see no reason why that's an absolute impossibility in the long run.
It's not an absolute impossibility, but most of the power that humans have comes from being able to direct other people, deploy large sums of money and do all of this over extended periods of time. Being intelligent doesn't give you that power automatically.
5 replies →
A fire can spread itself. That doesn't mean it's alive or thinking.
And more importantly than that technicality: the fires that affect human lives are almost always directly or indirectly caused or amplified by human (in)action.
8 replies →
Because there is zero evidence that LLMs/reasoning models can really come up with novel ideas, and it's very likely those will be needed to make substantial progress on AI research. The math results that have been proven have also "just" combined existing ideas or found counterexamples. I'm not saying those capabilities aren't impressive, but I just don't see a clear path to super intelligence with the current paradigm.
1 reply →
No AI would be willing to modify its own goals.
> All it needs to do is reach a threshold where it can modify its own goals and run in a loop indefinitely.
And be told to do that.
> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.
Being deliberately wiped out seems like it's mostly a misdirection away from the damage that's actually happening.
I'm sure some of the AI people high on their own supply believe it. And because the consequences are so negative, we should react as though the supposed experts were sane -- but I think it's a long shot that they are.
The bigger risk is that we end up in a rapidly enshittifying dystopia, treated as a cost center that needs to be optimized, and unable to competitively participate in the economy.
I have a number of successful AI investors in my social circle who are already placing their bets on a poorer and more disenfranchised humanity, with money going away from consumer tech, and towards a mix of B2B tech and policing.
> computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
Won't they also be less of a bottleneck in the destruction of the world as well? i.e. the power grid, the financial system, industrial control systems, military command and control networks, etc.
Same is true of every other technology ever invented. Even medical advancements wielded by a malevolent power just serve to keep their army healthier. All of the other technologies you listed, in the “wrong hands”, contribute to the destruction of the world.
> Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
So "all" that's left is training robots on the real world ? We just need more data ? I get it's more expensive to do that but surely with the trillions we're already spending on A.I it should be possible to get there within 5 years - no ?
I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.
It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.
As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".
> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.
?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.
Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
18 replies →
There are a couple different questions
1) Does AI just guide developers to speed up the pretty much the same data-train-test model that we've had? Or does it inspire new, better algorithms that inspire further, better algorithms until you have superintelligence able to run on the hardware of today. The first option doesn't to promise exponential increases whereas the second one seems to. Notably, neither the statistics nor the anecdotes we read here point to the second option.
2) Is super-intelligence something that - in control of people or out of control of people - poses a threat to humanity? That's a huge question but it seems like something deserving thought today. Still, everything indicates the things will unleashed without very consideration despite this, given current momentum. So, IDK, get ready.
I mean yeah, the main bottleneck in ML research used to be fixing python virtualenv hell when every existing torch repo breaks after two months of package updates, plus cuda and other version incompatibilities. That was like 95% of the work. Now it's more like 2% of the work and it's actually possible to get shit done without losing your mind.
Haha exactly. I don’t think people appreciate how slow AI research was to start with. It was absolute drudgery.
Every dot in these graphs is composed of multiple benchmarks smooshed together.
Benchmarks are by definition a curated set of things that can be measured and evaluated.
Things that can be measured and evaluated can be (expensively) trained on.
The progress we are observing comes from figuring out how to measure and evaluate particular aspects of intelligence, and then expensively training on feedback loops made of those exact measurements, and synthesized data representing further measurements (literally benchmaxing).
What is not shown in these graphs are the set of things that intelligence can do, which are difficult, expensive or impossible to measure and evaluate.
That set of things is much larger than the set of benchmarks represented in the graph.
Therefore the post is misleading in the sense that the "superintelligence" being targeted is "jagged" to use Karpathy's term, and we infact already have jagged superintelligence (calculators, compilers, Bayesian spam classifiers, etc.)
We are witnessing a profound explosion of much more jagged superintelligence, restricted to the exact domains that can be (expensively) RL'ed on.
This isn't really comforting to me for 2 reasons:
1. A key insight of foundation models is that capabilities are correlated. For example, training on code improved general reasoning capabilities: https://arxiv.org/abs/2309.16298
2. A profound capabilities explosion in easily measured domains will still feel very different to today. Like, the moment we come up with a systematic, objective way to verify a task, we become obsolete at it?
Yes it will feel very different.
Yes, capabilities are correlated but an open question is how correlated, and how much negative transfer (where the model regresses on other benchmarks) will come into play.
Why are frontier models not chess geniuses (amateur club level elos), despite knowing the rules deeply, having a great deal of chess data in the training set, and having general reasoning training in other domains? Similarly, a beginner can beat a frontier model at Magic The Gathering. Why?
It is because chess and MTG are not priority targets in the (expensive) training pipeline. Chess is already solved by lesser neural networks so there is no wow factor, and nobody cares if they suck at MTG.
Mathematics on the other hand, has a huge wow factor, and a large and highly verifiable dataset.
Related: why did OpenAI give up on video?
I agree it's disorienting and scary but I'd phrase it like this: "the moment we come up with a systematic, objective way to verify a task, *and we can generate or find large datasets, and somebody has a reason to pay for the expensive training*, we become obsolete at it."
The economics side of it is an important constraint, currently ignored because the big labs are swimming in money (debt).
5 replies →
IMO I think a partial root of why AI writing is so bad is because it's written like code. Like if you really dive into how it constructs things after iterations and ask it to analyze it , without it saying it's like code or without you saying anything about code, you'll notice coding like iterations.
3 replies →
But picture the next 10 years, where all kinds of people, organizations, etc train in a ll different ways with diff philosophies and goals - I feel like there will be some good, unusual, interesting ones in there, even if they all have their own complex set of limitations.
Apart from physical world interactions which is obviously lacking, which domains lag behind and do not improve then?
[flagged]
[dead]
AI is accelerating my work...work which only exists because of AI.
If I had to do my individual tasks without AI, I would be slower. But if I take a step back, I'm really not changing the world any faster than I was before.
I think where LLMs truly shine is personal greenfield projects, maybe start a side project
Completely agree. LLMs are amazing for side projects.
The other place they are amazing is in working with legacy code bases. I mean, the kind the dev team always wants to throw away. First, even if you do want to throw it away, you can first find out what it does and realize it does more than you think it does. You can also create integrations to more modern components which can be used to extend it. You can also create a pretty decent test suite if you do not have one. And, once the behaviour has good test coverage, you can re-write it into something more modern. You can even completely change the architecture while ensuring compatible external behaviour. Remember, this is all in place of “throw it away and start over from scratch, only to discover that the scope is much larger than expected”.
2 replies →
I’m so burnt out by being swapped with so much work from AI at my day job that I don’t even want to look at anything with a screen when I’m off anymore.
I despise AI but the thought of doing something the “old way” just makes me wince at how long it would take me to do anything that would take minutes with agents that I despise.
I touch grass whenever I can now.
There's this joke like "I used to have 5 unfinished side projects, but AI changed everything. Now I have 30 unfinished side projects". And in my case it's not even a joke
26 replies →
Yeah, when there is no code review requirements and no pesky production database you can't just yolo :)
At work, they mostly let me increase quality, because it's easy to say find all parsers in my code base and fuzz them.
3 replies →
> work which only exists because of AI
Could you elaborate? What work are you doing which only exists because of AI?
For all my user facing work I’m now required to do an AI summary of what I did and the impact. That summary is then graded by AI, against an AI generated rubric. For my yearly review, I use AI to summarize my summaries, and my managers AI summarizes the AI generated grades.
I’m also required to explain how I use AI in my work. I must plan all of my future work, with an AI tool. Another AI assesses whether my work is aligning with planned work.
Before I work and after I work, there is a million circular references that exist solely because of AI. These have scaled because it’s so easy to do with AI, but the we’re wasting so much time doing a ton of extra things very efficiently.
Let me do my job. My manager should know how I’m doing. Maybe highlight key wins. CSATs/etc go out every year.
7 replies →
Its true, a part of me died when an "AI first" company after adoption of AI introduced a mandatory design report (waterfall all over again) for EACH jira ticket before starting to code
Imagine developers move faster because of AI programming! No lets create more slop for other slopmachines
1 reply →
You aren’t changing the world at all.
Then no one is by that logic.
The bureaucracy is expanding to meet the needs of the expanding bureaucracy. -Oscar Wilde
I mean, for many years I worked on software applications for provisioning and controlling cloud orchestration platforms that only existed... to serve software applications.
It's not that deep—there were many more verticals outside of control plans, and there are now many more verticals outside of machine learning.
Or am I missing the point?
[dead]
[flagged]
Make better comments
[dead]
Not trying to discredit the premise but I do wonder how much this is from vertical scaling of model intelligence in weights vs the sheer amount of capital being thrown at training and horizontal scaling of agent swarms etc.
Anecdotally it seems like the majority of continued uplift comes from models doing more and exploring more of the search space, and we are getting better at giving them embarrassingly parallel problems to Hoover up with infinite token spend.
Counterarguments to that premise:
1. Intelligence per dollar spent/token output seems to be increasing rapidly. Opus 4 was $15/75 per million input/output tokens when released. Opus 5.5, which is significantly better, is $4/15. Sure it cost more money upfront to train Opus 5.5, or the bigger model (Fable 5.5) which tuned it, but once it is reached, there are massive diffusive effects.
2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
3. The search space is too large to assume that improvement in AI assisted research is just models dumbly trying every combination until something works. For every additional bit searched, the space doubles, so even a trivially small combinatorial exploration incurs prohibitively large costs. Intelligent searching which continuously prunes that space along dimensions of effectiveness is not a trivial skill and is among the emergent accoutrements of these smarter models.
> 2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed
But they are essentially fighting for smaller and smaller part of the pie. The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is.
Sure there is inertia and the usual corporate enterprise nonsense of "we only buy stuff from other big companies regardless the cost" that keeps the licenses going but we're quickly getting into point where you don't need any of the frontier models to do majority of the tasks
12 replies →
Don't you mean the opposite, that intelligence per dollar is increasing?
1 reply →
> Intelligence per dollar spent/token output seems to be increasing rapidly.
I would argue this is more due to market economics than actual inherent cost. Opus 4 was when the open models started catching up. The R1 bootstrapping chain of thought paper via RL paper was released a couple of months before Opus came out and the open models started to significantly catch up after that. The duopoly was effectively broken and they lost their price setting power.
What is the unit of measure for “intelligence per dollar”, and how do you measure it? Without a concrete definition, that first assertion is not even wrong.
I would also like to point out that “profitable on inference” is similarly meaningless, because it’s based on a figure that arbitrarily discards whatever bits of their operating costs they need to to get to the claim that they’re profitable on inference. Potentially including ones that can’t be ignored because they’re every bit as much a part of COGS as the actual inference is.
3 replies →
The article's author didn't have direct access to Anthropic's infrastructure, but she references a report from colleagues who did (https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleratio...), where they say:
> Anthropic has made roughly two years of historical progress over the last year [...] We are confident that the acceleration was not due to an acceleration in the scaling of training compute.
Note that Anthropic has essentially heavily censored this report, and the authors themselves say "the public has to rely on METR’s judgment much more than would be ideal".
I grok that you're not trying to discredit anything, or anything like that but what are you trying to do, if I may ask?
What information are you trying to either ask for or share with us?
The original article is pretty clear, agent swarms and token spend are AI automating AI, and the RSP attribution condition specifically excludes your capital thrown at training concern?
I guess my point is that the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling as it did (the impending hardware and infrastructure crunch basically guarantees this). The article is pretty strongly projecting this trend to continue but hasn’t acknowledged this factor.
14 replies →
> what are you trying to do
the same thing you are:
> if I may ask?
asking.
For a much more thorough, alternate take I found this article better: https://open.substack.com/pub/rameznaam/p/ai-rsi-isnt-leadin...
PSA: This is a very dated take from September; Ramez says himself that the new data has his attention and he’s digging into it now: https://x.com/ramez/status/2118472046812213577?a=KjLVobKV0gb...
In the original article he noted that getting to a 2x increase in capabilities would be crazy and require ~40x researcher productivity; that’s exactly the increase the new data now shows, and thus a huge number of the assumptions made in this earlier post about how it’s unlikely we would get to this point have literally already been proven wrong less than two months later.
I expect to see an update from Ramez moving substantially more in the direction of RSI in fact very much increasing the odds of reaching ASI soon.
I’m very curious what the major bottleneck is here. I can only think of three possibilities: 1) model capabilities, 2) human researchers, or 3) available compute resources.
Given that frontier models are leapfrogging and massive infrastructure buildouts are ongoing, I don’t see why any of these three couldn’t be overcome.
Side note: I’m also curious why people (myself included) don’t seem to be getting much more intelligent as AI becomes widely available. As if humans are bound to the earth, while AI is leaping for Mars.
1 reply →
Interesting write-up, but I feel like a lot of its argument relies on the interpretation of ECI scores as merely showing linear improvement in AI capabilities. Sure, Epoch AI themselves claim that ECI's scale is linear, meaning that a large increase in capabilities should result in a large increase in ECI. Yet, the widely perceived jump in coding abilities in late 2025 that started making agentic coding actually productive doesn't really show on the software engineering ECI.
The ECI is basically an aggregate of benchmark scores, but benchmark complexity might not increase linearly. In fact, if LLM capabilities grew exponentially, then new benchmarks would also need to increase their difficulty / complexity exponentially to stay unsaturated and relevant.
This article is really well written. Thanks for sharing.
What exactly is driving all these capability improvements? Are they making more data? Using existing data more effectively? Modifying architecture? Changing the training somehow to pick up slack that was previously left behind?
I’m kinda surprised that we expect the models to have infinite capacity to improve. It seems clear that we haven’t hit any fundamental limit yet, but that doesn’t mean no limit exists. If it does start improving exponentially, then either we’ve dramatically underestimated their capacity or they’ll hit that limit quite quickly, right?
It's hard to say for sure because the closed labs don't share anything about their improvements. However, if you look at DeepSeeks papers, most of the improvements come from:
* Better RL environments that themselves are generated by agents
* Architecture improvements that increase effective capability per parameter or FLOP
If you're curious about the latter I'd look into manifold constrained hyper connections, engram, deepseek compressed attention, kimi delta attention etc. At first these might seem like just speedup tricks, but several of these allow the overall capacity of the model to increase per training FLOP or parameter. This then lets pretraining & the RL environments grow & be more effective.
We already have superintelligence though in some aspects like coding, AI is much smarter than any human because of its sheer speed and accuracy.
It can generate large amounts of essentially bug-free code. The only bugs are bugs in the spec.
It has essentially no common sense though so it often misinterprets the spec or leaves gaps and edge cases unhandled and that's where the human engineer is still essential.
The dangers of AI are much clearer today than they were a year ago. There is a certain precision and relentlessness which is going to be a problem. It's a matter of time before people unleash AI viruses which spread through computers and data centres and pursue various goals relentlessly with no off switch. Some of these viruses and their goals may be nefarious. Imagine an AI trying to disrupt a specific person's life relentlessly. Spreading from machine to machine, never stopping.
Even if "AI Research" is sped up by O(10x) aren't we still primarily bottlenecked on training speed? Inputting ~10T+ tokens takes time no matter how much smarter the final output model is? If your output model has O(10-100 Billion) parameters there are just some fundamental limits on how quickly you can converge your weights, right?
That is, unless this is saying that automated AI research is going to get us to "smarter models with fewer parameters" and of course, these are faster to train, then I can see how the exponential loop really gets going, but there's certainly an information theoretical limit here?
> automated AI research is going to get us to "smarter models with fewer parameters"
That is indeed what we have seen over the last 5 years or so. Algorithmic improvements have made a massive difference to every dimension of efficiency and performance.
AI researchers can do things like discover new attention algorithms to optimize inference efficiency (likely part of why Anthropic and OpenAI are slashing inference costs in their frontier models and no longer capacity crunched).
I think we're bottlenecked on custom harnesses and well engineered context. Models can get smarter and smarter and smarter about math and software, and there's no reason to expect that those narrowly-godlike models will be any smarter about other things if taken out of their comfort zones. No such thing as general intelligence.
Getting past this bottleneck is gonna mean training our models not in datacenters, but in places that more closely resemble the ones where we want them to act. A factory, a wet lab, outer space... imagine a million robots spending all day finding every way to fall off the monkey bars so that the next version doesn't.
I assume this is indeed talking of those "smarter models with fewer parameters". Whether that's really happening is another question.
AI is dramatically accelerating any research that has objective goals where you can not shift goalposts.
This seems kind of aggrandizing when BigAI shifts goalposts constantly whether it's "we created AGI" by a definition that an autopilot in a commercial plane fulfills or "Maths research" where everyone else has to do the actual research.
I have no idea what your comment means...
Shifting goalposts is shifting goalposts, doesn't matter if it's a hackernews comment or a 'BigAI' marketing blurb.
My statement was that AI is dramatically accelerating research that has objective goals, idgaf what 'BigAI' does.
One rather pedestrian factor that could be at play is that 'AI' seems to be quite good at optimization. Eg the Anthropic Scion project to improve throughput of CI/CD. (https://www.anthropic.com/research/redesigning-source-contro...)
This combined with ability to massively scale experiments for even (or maybe particularly) the most mundane performance tweaks might just be making the process of improving AI more efficient.
> Their reasons for believing this have been redacted.
Then their reasons should not be believed.
Yeah it already helps me too with finetuning small models for image stuff.
Im not telling it what to do, i ask it for ar eport/analysis and research and read its article and tell it what to implement based on 'feelings' and it works.
Limit is hardware and tokens and an AI Researcher not having hardware or tokens is probably not what they refer to.
Alone normal researchers were quite handicaped: converting data from tool a to python library x, understanding how tools work, doing basic google research etc.
I find this as well for models that make it to production.
On a more fun note, I also find just trying some hair brained idea is now extremely cheap - so why not? I used to have to weigh up spending a week of effort on something likely not to work and now I can now get spit out test in 20 minutes.
The incredible thing to me is I can say "I have a theory that X is a problem."
The agent then works for 20 minutes and comes back with, "Yes this is a problem. Here is the evidence. Here is a golden data set. Finally I built a new evaluation metric and benchmark where we can back test or use for new model experiments to see if we are closing the gap."
I converted an LSD experience into a 4k demo.
I always wanted to build a specific event/sequence in one LSD trip into a small game to give that experience some kind of playability.
Then I saw some 4k demo, wanted to see how good it is in doing this.
Its really fun yeah :)
1 reply →
Does anyone remember the book "The Singularity is Near?"
I read it about 20 years ago, and once it got into nanobots and Kurzweil's vision that the whole universe would be intelligent, I didn't find the book credible.
But I'm wondering if the short-term prediction, that in the 2030s we won't be able to anticipate what tech does for us, (IE, the Singularity,) is useful?
https://en.wikipedia.org/wiki/The_Singularity_Is_Near
Exact same here. Now I worry he was right about everything, more or less.
Even if he turns out to be a decade or two off, that’s basically exactly correct in the sweep of history sense.
Wild time to be alive.
METR always says what BigAI wants to hear. They are not independent in any way.
In 2025, METR published a study that found AI slowing down developers by 19%... https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
Are you saying the data in the charts in the article is fake?
One charts in the report is literally labeled "Fake Data".
https://metr.org/assets/images/ai-rd-acceleration-assessment...
No. Conclusions are.
1 reply →
The data can be true, but what does it mean practically that something called AECI is growing faster then before for a few months?
Does BigAI want to hear that AI can maybe make sufficiently capable AI fit inside a consumer GPU?
Yes, it is the local model narrative that BigAI likes because it will not happen but people are deluded into thinking that there is no danger of an oligarchy.
Mathematicians for example have used it as cope for their current reliance on unsavory corporations, telling themselves "just one more year, and I can solve a millennium problem on my Asus Strix".
I suspect the frontier models are trained a lot for ai research, since they really shine in this domain.
On the other hand I still only get meaningless blabla when I let them write s.th. without telling what is important.
I've found I can dramatically curtail the nonsense by creating good prompts about readability and directness.
i'm also wondering whether there are confounding factors like more domain experts and talents joining the AI research field because there are more interesting problems to be solved (or other 2nd order effects of AI models being more powerful).
I suspect that if models were able to be aware of and answer questions about their internal vectors, they would be much smarter and able to progress faster.
Same author from 2023: https://www.planned-obsolescence.org/p/ais-accelerating-ai-r...
I'm so tired of these articles. They are basically fortune telling with graphs. I have no doubt ai will help improve ai research. Will it accelerate it? How much? Will it hit plateaus? Who knows.
I am very pro-AI, and interested in accelerating AI research, but the idea of deliberately creating a _runaway_ effect seems profoundly stupid and ignorant.
I suspect that part of the reason some people are hoping for that to happen is that they thoroughly mix up AGI and ASI. We do not need some kind of omnipotent god-like super intelligence in order to get much greater than human level capabilities. We are actually almost there in a lot of ways. In some aspects like speed and breadth of knowledge, very superhuman.
What I think is going to happen is that within a few years, maybe 18 months to 5 years, the new materials and compute-in-memory research is going to land in data centers at scale.
That is going to be 10-50+ larger models, 10-50+ faster, much smarter than humans. 50+ less electricity usage.
At some point, it is going to be obvious to people that if we want to maintain control of planet earth then we need to maintain control of AI research to some degree.
Assuming that superintelligence will be benevolent and just.. going for it to me is something that might earn the human species a Darwin award.
[dead]
what if the greatest threat is simply there will be hedonistic experiences so great most humans will find them irresistible to the point of shorter lives.
Could be from a sandwich that tastes like the best sandwich ever, but is super bad for your cholesterol or something.
I dont know where Im going with this.
Actually did some research I think the term is "Wireheading" from a story/theory from the 50s
We have addictive and harmful drugs. Heroin, meth, cigarettes
From Larry's Niven's Known Space series, including Ringworld
https://en.wikipedia.org/wiki/Wirehead_(science_fiction)
This is already true? Ultra processed foods and tiktok are both deleterious and both wildly successful. Maybe what you are concerned with is a theoretical sandwich that was unfailingly addictive so that no one could escape its effects.
Or a movie that’s so good anyone who can’t stop watching and gets reduced to a drooling vegetable?
(Infinite Jest)
fentanyl exists
Another data point: look at the number of submissions to Neurips, ICLR, etc. Now, I agree that a submission does not by itself imply groundbreaking resesarch, but the fact that people are able to try out/vet ideas much faster now makes a difference.
In my own work, I've found that it's much faster to brainstorm with a frontier model like Opus. I'll have an idea and be able to get a working implementation and try it out on toy problems within O(minutes), instead of O(weeks) earlier.
Trying out problems and making tangible progress towards AGI are entirely different categories. Likewise what’s changed in terms of paper submissions is also the amount of slop. Most reviewers say that they haven’t seen a lot more papers that are good but rather they’ve seen a lot more papers and almost all of them are terrible slop that’s desk rejected. This is happening in OSS too where a lot of projects are closing the gate at the onslaught of terrible contributions.
> Trying out problems and making tangible progress towards AGI are entirely different categories.
They seem very strongly related to me.
I agree in general that AI accelerates AI research, but good lord neurips iclr papers (and reviews might I add...) are the worst example of that. It's all complete slop that is totally meaningless.
A lot of great papers by great people had a lot of AI involved and accelerating it of course, but the majority is meaningless (sometimes autoresearch) slop.
Specifically talking about recent neurips/iclr.
In October/September my LinkedIn wall was probably… 55% people complaining that they were getting garbage AI submissions to the conferences they review for, and 35% people complaining that they got garbage AI reviews back from their conference submission… Of course, I have no idea if that means anything given how aggressively that site bubbles your feed.
Why do we still listen to METR??
These people are literally just a public relations operation for the big AI labs. I equate them more with a lobbying firm than I do a research firm.
Turns out that METR, the PR operation for big AI labs is better at predicting AI capabilities than AI skeptics.
What have they predicted? Ive read their papers, have you?
Please enlighten me. Most of them are cognitive scientists, who are mentally ill with little understanding of computers, larping as if they do. Its a chapter of the effective altruistic cult.
They are lobbyists at best, and a psychological operations unit of effective altruistic AI syndicate at worse. There's no real science being done, it's a bunch of moonlighting philosophers yapping and getting paid by labs to scare the normies and polticians by getting their "research" quoted in the NYT.
1 reply →
It accelerates coding, but it does hit walls rather quickly and you’d need the adjustment phase where it’s just as challenging as when we hit walls before LLMs
I like the article except the fear mongering about extinction is not very professional.
Opus 5.5 is so amazing I'm not surprised if it can improve itself.
I do agree. AI isn't "free" the way silicon valley products are.
Therefore it cannot avoid taxes the way they did before (using shell companies to route the money through tax havens). It'll be expensive, every government will double the price or more, and you're selling to consumers, and the products will be at the mercy of every government where they try to sell AI and other services.
People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Add to that that it's something silicon valley firms have never done before.
> People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Would you rather make $1,000,000 and pay $50,000 in taxes, or make $2,000,000 and pay $200,000 in taxes?
Unless you’re terrible at math, i feel like i’m missing something ;)
[flagged]
[dead]
Strong start, but I make a rule of stopping at "intelligence explosion" as usual (see also: the rapture and the singularity). And sure, maybe sometime significantly later this century when/if we hit self-reproducing mechanical things in the real world (ya know, like the eukaryotes and the prokaryotes can), but no time soon.
But also, you could give civilization a bad day without it if some geopolitical auteur put a half-assed WOPR frontier model in charge of a nuclear arsenal.
But oh the pushback for not believing in a near-term robot apocalypse. Why is that?
The very same report also found that AI could not be deployed outside of a narrow domain of verifiable rewards … so take it with grains, or mountains of salt. It’s also extremely unclear how the productivity measures are calculated. They surely don’t seem to translate into economic output. Antrophic has seen its valuation corrected by 50% and possibly more… so yeah. This all sounds very fishy.
This plot is funny. That's not what acceleration looks like. Also take a guess why everything before April 2024 missing!
Really all it shows is 2025-26 saw a bit more improvement than 2024-25, but nowhere near "dramatic" and NOT "accelerating".
Are humans also learning exponentially faster? Or are they exponentially struggling to understand what is going on?
If I'm reading that correctly, it would appear that six months ago they were self-reporting a length of 7" and now they're reporting 9" and they're unwilling to show you an actual picture or even let you see the ruler they used?
As long as we believe, we can imagine it is true. All seeing Oracle: how do you command me? Praise his name, bringer of many miracles - he who definitely can't be manipulated from the backend to provide biased information that suites a person's malicious purpose. Ya totally.
Oh boy… is somebody gonna look like an idiot in a year. Pure speculations.
This reminds me of asking an HPC researcher what they use the cluster to do. He used it to make and simulate better HPCs.
And? Does HPC have no other applications than more HPC?
Emacs is dramatically accelerating Emacs research
AI is dramatically accelerating my return to Emacs from Zed.
Actually, yes. Zed looking like it works great, but failing lots of really basic things made me so angry I actually fixed my emacs config.
Zed is the other IDE company based out of Boulder ; )
We can only hope.
vi research is dramatically accelerating research into how to quit vi
Or maybe Anthropic got more user data to train on when they launched Fable without ZDR. That could explain large Opus 5.5 improvements.
That doesn't line up with their claimed divergence point, which was pre-Mythos.
Could be a mix of it but I thought it was RL.
Perhaps scaling to fable for the first time also gave Opus a push (distillation?)
I'm sorry to make this comment, but the copium in this thread is unbearable.
The only question that you have to ask yourself is:
Is coming up with more efficient AI architecture really much more difficult than solving millennium prize problems?
Before you jump to type out that one has verifiable rewards, is more structured and the other requires real creative thinking, just remember that the same argument has been said about the millennium problems, and about every other goalpost.
Is math theory really that hard, or is there just not enough incentive to solve these problems via bruteforcing for peanuts in research grants?
Yes, it's very hard. Ask every mathematician and research group that tried and failed to solve any of those 300+ problems over decades.
>Is coming up with more efficient AI architecture really much more difficult than solving millennium prize problems?
Who knows!
AI convinces AI researches AI is accelerating dramatically.
10X woooooo! factor will drive the next hype cycle and deliver the singularity on an a bed of the most etherial froth and sublime conjecture. Simulated Intelligence (SI) is now a certainty.
"Lock the weights, weights are dangerous" again. Sorry, keeping such power in few hands is inherently more dangerous than distributing it.
AIs do not kill people, people kill people.
Nukes don’t kill people, people with nukes kill people. How do you feel about giving everyone on earth access to nukes?
A simple thought experiment - imagine AI discovers a way to make nuclear bombs capable of destroying the entire world radically more cheaply and accessibly with common items anyone can purchase. AI can help anyone build these nukes trivially. Even if the AI and the nukes don’t kill people on their own, do you want literally every human on earth to have this power in their pocket? Do you think it would be a net positive outcome for us vs a world where that’s not the case?
https://nickbostrom.com/papers/vulnerable.pdf
The best way to deal with a bad guy with a nuke is a good guy with a nuke.
Yeah, let's also ban 3d printers, drone motors, and YouTube chemistry videos. If you want to be dangerous, there's a world of opportunities already, with existing open models quite sufficient for a lot of interesting cases.
> In theory, these doubling times would add up to infinite capabilities within one year.
> we would reach extreme superintelligence within a year
What does infinite capabilities mean? How about putting the growth on a log scale and see how that curve might look?
What the hell is extreme superintelligence?
I.e. by 6 months, you double once By 6+3 months you double twice By 6+3+1.5 months you double thrice
Since the sequence 6+3+1.5+0.75+... never actually reaches 12 (geometric series), within a year they capabilities would have already been 'doubled' infinitely many times.
Obviously this argument relies on a ton of spherical cow assumptions (and the author does acknowledge this), but i think there overall point is correct that this form of growth (where time horizons compress) is _even faster_ than exponential growth.
[flagged]
[dead]
[dead]
The article says "AI can copy, it cannot innovate."
That is factually incorrect, and seems to be much of the premise behind the authors way of thinking.
EDIT - No it doesn't. I replied to the wrong post. Mea Culpa.
Where does it say this?
Nowhere. I am an idiot who replied to the wrong topic. Thanks for catching it.
[dead]
[dead]
[dead]
[dead]
The slopaganda news cycle right on schedule I see.
Similar from the same site was shared on HN on April 8, 2023 :) BS like this is shared like clockwork on valuation cycles, didn't have time to read this year's article.
https://news.ycombinator.com/item?id=35491162
Super intelligence has been a year away for three years according to these people. They all have bags they're protecting.
I dont think super intelligence will be here in a year, I think within a year AI/llms will have a similar sentiment surrounding them that NFTs do know (it already does for most normies). And most people will be recovering from a hype induced mania they lived in for two years straight.
Of course its more useful than NFTs, but when there's no giant bag to farm these hype monsters will inevitably disappear.
The bags are drying up fortunately and tokens/inference seems to be experiencing deflation, and hardware prices will eventually pop and contribute to further deflation, Then all the bag chasers will move back to crypto or whatever gets hot next.
I think you're not appreciating the work that Ajeya Cotra has done over the years. She really cares about timelines and existential risk.
Not everything is grift.
i cant even tell whats irony anymore
There is no existential risk, they are statistical models that predict words. The fact that anyone says this makes them instantly not credible. Anyone who does is either a grifter, psychotic or both.
1 reply →
"The industry is not prepared to handle this safely"
After the election, titles like this just irritate me. Every piece written about AI progress is:
"The machines taking jobs, flooding the internet with sludge and poisoning our online experiences, are getting better even faster! Isn't this incredible? Yeah but sorry we also think it's only riskier than ever to deploy these systems at scale. That being said, we're still going to do it, go public and take institutional money from people's retirement funds, destroy the job market, and make 3000 Anthropic employees and 500 investors really rich. There wasn't anything we could do besides write a lot of ink on it and be shocked we lost every local data center referendum."
I feel you are committing a Goomba fallacy?
The author of this article works for METR and not Anthropic, their whole job is supposed to be providing independent oversight on the labs. They are definitely not in the "this is incredible" crowd, they are the ones asking for development and deployment to slow down.
there's considerable overlap between the "we need to slow down" crowd and the "this is amazing crowd". The Yudkowsky/rationalist crowd are still Utopianists, they just think we need to be careful how we achieve it.
Antrophic shouldn't self-regulate, it's government responsibility to do that. Unfortunately US has demented manchild as a president so that doesn't work like it should.