Comment by pyridines

A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.

But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.

I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.

I feel like there’s a weird conflicts in how folks mental model of LLM development.

It’s not the model I don’t trust, it’s myself. The model is wrong _all the time_ because - it’s easy to verify the code - it’s hard to verify that I knew what I was talking about when I prompted it.

So the idea that you can broadly speaking take the human out of the loop. I think suggests to me a level of consistency in the contextual environment that would probably never exist.

At some point it’s politics. The model can come up with a better answer than my boss, and then my boss can just ignore it. Taking the human out of the loop broadly speaking implies that we all agree on what we’re trying to optimize.

> If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail

But suppose some future holy grail AI can do much more than that.

Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.

But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).

Isn't it fun to imagine how life would look like in that scenario?

We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about

>Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say.

I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.

The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.

A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).

Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.

Nit, but Terrence Tao did not write this article, it’s a guest post.

LLMs dont create anything new, if programmers stop reading the code technology will be forever frozen to 2022, no new programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks everything will be based on the training data and future generations will forget about all the primitives we now take for granted.

If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.

> Terence Tao is arguing

It's a guest post by Amit Sahai, FWIW.

> Why check it? It's obviously going to be correct.

I used to obsessively check the assembly (machine code) output of my C++ compiler because it would regularly miss essential optimisations and my tight inner loops would blow through my time budget.

I haven't done this in decades, it's just not worth the hassle almost always, the compiler will do a better job than me at optimising the runtime performance. If anything, "second guessing" the compiler will often degrade performance!

I'm starting to notice this now, where my clumsy attempts at refactoring AI-authored code introduced bugs that the original vibe-coded output didn't have.

I think the disagreement here is in time scale. The human-in-the-loop period for software development may be orders of magnitude shorter than infrastructure engineering and other physical applications.

> Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing.

If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.

> What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?

For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.

Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.

Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.

How crazy will the world be once our [current gen] ANN are like that!

What an amazing thought.

> Why check it? It's obviously going to be correct.

Because as complexity floor increases, it's "going to be correct" in hyper-specific, hyper-literal, insidious ways, with 10-50x more lines of code than necessary, and tens to hundreds of incredibly useless tests that give the illusion of quality, and cause cascading effects where seemingly irrelevant and orthogonal features that were once working end up breaking because of the agent's changes

> doesn't convincingly justify why, in my opinion.

I'm convinced this agency argument is correct [for the next N months]. But yeah, it's vibes. And you could probably create a reasonable proxy measure for this.

So I wouldn't call his argument unconvincing, I would call it unformalized. In order to walk this world you're gonna have to contend with some informal arguments that are powerful, correct, and should be convincing.

Terence Tao did NOT say that. Amit Sahai did. Maybe you are just being less careful?

The problem with a terawatt fusion plant isn’t that the first one will be broken, or the tenth one in some other way. The problem is the hundredth will work flawlessly, and so will the thousandth, and a petawatt is serious waste heat to reject; if we keep building them on this planet, we’ll all simply cook.

Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.

I think a key flaw in this reasoning is that you extrapolated code needing fewer edits to AI generating entire fusion plants in one shot (in the sense of being instructed once), mostly glossing over the long intermediate period where AI will need significant back and forth to do such things. At the simplest level, it'll need to ask for planning new experiments, experimental results, test runs, etc.

Since these resources are still extracted and allocated by humans, humans will need to be able to take apart what the AI produces, and if we want to scale this capability, we're going to need many more researchers.

This is a guest post by Amit Sahai.

>Terence Tao is arguing

This is a guest post by Amit Sahai.

it's incredible the amount of people who think all these recent posts in Taos blog were written by him.

Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.

Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.

Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.

> I guess, the possibility of reclaiming human meaning and purpose.

As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.

The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.

I feel you. We take a risk when we check our work less and we own the consequences. That's not new, and it doesn't make us bad. It's just the human condition from time immemorial. The solution is not obsessively combing through every detail of our work, it's better monitoring and control on the places where mistakes can have an impact.

In my experience, LLMs are becoming very good at executing, but not a creating novel ideas or being creative.

Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.

Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.

For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.

You oppose correctness to meaning and purpose, which you seem to imply are impractical values. (Worthless values, then?) But you don't mention creativity. The article blithely says that AI creates new ideas and understands things. I don't think it does.

I'm not sure the scarce resource will be people capable of understanding the AI's work. It may be institutions willing to tolerate the cost of understanding it

Please read the post. It's not written by Tao.

Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.

There's a lot of languages where that's nowhere near as simple as you make it sound. Floating point, decimals, etc.

Then there's other little quirks like rounding rules: https://en.wikipedia.org/wiki/Rounding

Basically, adding numbers together is exactly the sort of thing AIs still muck up spectacularly, precisely because they either fail to understand the context of the problem, or fail to ask about an assumption they make.

That you've had so many replies and no-one else has even mentioned this is in itself worrying.

Your own example proves your point is wrong.

The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.

I don't think models have improved in the "not needing direction" part, at least not proportionately to their other improvements.

Math problems and computer programs are two places where a model can get it's direction from the problem itself. Mathematics may well be larger than just problems.

Correctness is a scale, and what is correct can become incorrect with enough sliding of said scale over time.

Because "the cause of the cause is the reason".

Probably human accountability.

Hypothetically, if a system built by humans then helps humans build the next system that is then initially kicked off with "design something that may influence the lives of other humans" and we all write down that AI is really good so inductively we thought itd be really good at the next thing it builds, and then a critical error is introduced and does "insert tragedy that you personally care about" then your rage would cause you to act politically and want to ask who signed off on it. If the engineering costs outweigh the fine then yeah thats what's probably going to happen but from a human accountability standpoint thats going to suck for the unlucky ones

The question is simply “will there be people who, having never gone through the fundamental steps of trial and error, learning, etc. are actually ABLE to understand and verify what the LLM is proposing AND be able to see potential pitfalls/design processes and failure modes should the worst happen.”

This isn’t a question of “what can an LLM normalize”.

If AI ruins humans doing mathematics because of "economic strategy" we should destroy, not the machines (although the data-centers will be burned down as a byproduct) but the economic system that demands this.

> I pored over every single line of code Claude generated with razor sharp scrutiny.

That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).

> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.

I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.

> Why check it? It's obviously going to be correct.

And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.

All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.

>the economically dominant strategy to not verify them and not double check them

In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".

The models get things wrong in the way that humans don't.

They will never make a logical error yet make terrible assumptions and poor long scale decisions.

Wake me up when an agent swarm can write gcc in a box sealed from the internet.

> my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.

I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.

The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".

>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?

Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?

Am I the only guy who still thinks we're kind of putting the cart before the horse here? Look, I would love to live in a world where AI is in the business of designing terawatt fusion plants and revolutionizing all other aspects of society. But right now it can't even really tell a puddle in the road. I feel like we have a really long way to go here, hype-laden PR releases about solving math problems aside.

We’re seeing more and more slippery slope arguments, except the slippery slope leads to human cognitive oblivion and it is actually a good thing actually.

> I just hope that there are more Terence Taos out there than people like me.

Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?

This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.

People please, an LLM is just a vector database that spits out statistically viable answers which highly depend on its training material. There's no real "intelligence" involved.