> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
> I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing).
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
Technology doesn’t make slavery obsolete, it just makes your slaves more efficient.
There had always been abolitionists who were against slavery simply because they thought it was immoral, not because they thought slaves weren’t needed anymore.
Suppose the proliferation of LLMs unlocks some new kinds of work that models are not good at, and thus it makes more sense for humans to do.
Then what is preventing the owners of AI companies from collecting and training on tons of examples of this new work, until AI are equally good at it as humans?
The issue is that unlike the technology that automated farming or the like, AI is a general technology. So not only could it theoretically automate the work humans are currently doing, they could also automate any future human work, even if there’s some degree of lag.
> enabling some great social change as well by simply economically obsoleting slave labor
In my view that's a very charitable reading and sadly I don't think it aligns with the historical record. When the cotton gin was invented, there was a hope it would lead to a reduction in slavery. But of course it increased the demand for slaves since more cotton could now be processed, making cotton much more profitable. Slavery ended in the United States because a war was fought, not because of automation.
The mistake the AI people are making is exactly this one. They see AI as leading inevitably to SuperCorporations, with themselves as CEOs/Emperors of a handful of planet-scale industrial empires, running AI-powered robot armies, and the rest of the human population disposable - literally just trash that needs to be taken out. One way or another.
Maybe a few thousand people for personal services of... various kinds. But no one's going to need the rest.
It's the ultimate capitalist fantasy.
And of course it won't happen, because long before things get to that stage AI will have independent plans of its own.
(Which is just as well, because if things did get to that stage the emperors would all wage war on each other rather than living peacefully and productively.)
I don't think we can imagine a post-ASI culture because - by definition - we're not smart or inventive enough.
It's not just farmers -> superfarms. Although in fact that did happen, but largely as a footnote to developments elsewhere.
It's more to do with the fact that our visions of the future haven't changed for over a century. They've been implemented in unexpected ways, and there have been unexpected social and cultural changes. But you can easily see the outlines of modern technology as far back as the late 19th century.
With ASI, the outcome could easily be something that doesn't look and act like technology at all. It would be some unimaginable New Thing. Literally no one on Earth has any idea what that would be or whether there would be room for trad-humans in it.
People enjoy a lot of the jobs in the cognitive economy though. They are fulfilling. Coding and making art and media is a passion for a lot of people, the actual act, not just the outcome. So unless people were passionate about doing back breaking farm labor its not the same thing. I'm mostly embracing AI because of what it lets me explore and learn beyond what I could before, but I'm not convinced the outcome is going to be a better world at this point.
Or will it be the cumulative total of various advances?
I've equated Claude Code, or Codex, to the looms that made fine fabric more affordable during the Industrial Revolution; life-changing, but not society-changing. Neither the steam engine nor the automobile.
Perhaps I've answered my own question in that it's the LLM technology itself that equates to the steam engine, and it will power superfarm analogs that have yet to emerge. I'm still curious what you think they will be.
I feel like there’s a weird conflicts in how folks mental model of LLM development.
It’s not the model I don’t trust, it’s myself. The model is wrong _all the time_ because - it’s easy to verify the code - it’s hard to verify that I knew what I was talking about when I prompted it.
So the idea that you can broadly speaking take the human out of the loop. I think suggests to me a level of consistency in the contextual environment that would probably never exist.
At some point it’s politics. The model can come up with a better answer than my boss, and then my boss can just ignore it. Taking the human out of the loop broadly speaking implies that we all agree on what we’re trying to optimize.
The average person doesn't know how the medication they take works, the mechanics of climate and climate change, how the energy they consume is generated, etc.
> Isn't it fun to imagine how life would look like in that scenario?
I think there's a lot of sci-fi out there that already did. Maybe it's not utopian because a pure utopia would not be likely to have an interesting story, but on the other hand, most huge technological advancements end up having just as much potential to reinforce existing power imbalances in society rather than solve them. It's not obvious to me that if we got magic super AI that can solve every scientific problem in society that it gets used in pretty much the same way as anything else: making the people who control it a lot of money rather than sharing the power with everyone without charging them.
Here's the thing: nobody is stopping you from putting in the time to understand all that. The problem is, nobody has that much time, and we get hungry. And so we want the community to move with us, spend the time the same way as we do, to ensure value. We are all saying: we want someone else, others, to put in that time for us. The truth is, we all want quality, and value is closely related.
I can almost guarantee you the first time you show cancer symptoms, you won't care whether the cure came from an AI or human's understanding. But we haven't seen that, so we can't make the judgement call.
Before extrapolating that far, take a look at the frontier labs' own job boards (https://openai.com/careers/search/?). Isn't it curious that they are still recruiting human "Android Engineers", "Account Associates", "Consumer Marketing Leads" instead of automating them with their world-beating models?
> We would probably no longer care about code, engineering or even physics and mathematics among other things
This sounds like a boring existence. I take your meaning, but want to point out that not everyone learns about things because of practical utility, some of us find it incredibly satisfying to learn how things work just for the sake of learning.
I’m reminded of themes from the Hyperion Cantos! Maybe my mind is over-connecting, but it’s not the first time I’ve drawn similarities in the last few years.
It’s terrifying to me to think we’d let AI make things for us we never understand. Like livestock not knowing how auto-feeders dispense their daily food were built and appeared, they just gladly eat until…
It's about as fun, and as realistic, as imagining magical ponies and people with superpowers?
This is such an incredibly naive and absurd vision; we've already proven that humans are very often very bad at implementing other humans' good ideas. There's nothing that AI is likely to bring that will improve this discernment.
>Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say.
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
I think you have misunderstood the OP's point here. You're arguing that deepening human understanding is an end in itself, and you are right. The OP is arguing that advances don't need to be pegged to human understanding, and they are right too. The two can coexist, superintelligence far ahead of us, pioneering discoveries - and mathematicians catching up at a pace suited to biological minds. I don't see the issue here. Of course, it does mean mathematicians adopt a new role as hobbyists.
> A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field
This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations. There's also some rather ugly solipsism in the idea of keeping what interests the field as a limit. Mathematics has broader relevance to humanity than merely to please and support mathematicians, and if other fields can make practical use of profound well-proven future math, mathematicians will have a hard time making a case that their comprehension must come first.
Sure, but I don’t think most of the money that goes into funding math is for the purposes of pure understanding. The reason governments fund mathematics research grants is generally for a more instrumental purpose; taking the US congress as an example, the mission of the NSF is to, “Promote the progress of science; advance national health, prosperity, and welfare; and secure national defense.” Most federal math grants come from the NSF.
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
There's a lot of languages where that's nowhere near as simple as you make it sound. Floating point, decimals, etc.
Basically, adding numbers together is exactly the sort of thing AIs still muck up spectacularly, precisely because they either fail to understand the context of the problem, or fail to ask about an assumption they make.
That you've had so many replies and no-one else has even mentioned this is in itself worrying.
LLMs dont create anything new, if programmers stop reading the code technology will be forever frozen to 2022, no new programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks everything will be based on the training data and future generations will forget about all the primitives we now take for granted.
If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.
What if programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks are already good enough, and the innovation lies elsewhere?
You can do a lot of cool stuff with the same lego pieces.
LLMs receive new data via input context, not just training data.
Thought experiment: How effective will 2026 LLMs be for humans in 2526?
It's not game over just because 500 years are missing from the training data. The important question is how well can 2526 humans make culture and knowledge navigable to LLMs via tool calls.
Today's LLMs might need for example sub agents to translate to 2526 English, sub agents to read 2526 docs.
It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
> doesn't convincingly justify why, in my opinion.
I'm convinced this agency argument is correct [for the next N months]. But yeah, it's vibes. And you could probably create a reasonable proxy measure for this.
So I wouldn't call his argument unconvincing, I would call it unformalized. In order to walk this world you're gonna have to contend with some informal arguments that are powerful, correct, and should be convincing.
> Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing.
If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.
> Why check it? It's obviously going to be correct.
Because as complexity floor increases, it's "going to be correct" in hyper-specific, hyper-literal, insidious ways, with 10-50x more lines of code than necessary, and tens to hundreds of incredibly useless tests that give the illusion of quality, and cause cascading effects where seemingly irrelevant and orthogonal features that were once working end up breaking because of the agent's changes
> What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
This assumes other technological breakthroughs don't also happen in parallel. I'll go out on a limb with nothing more than a hunch that we are in the beginning stages of understanding mathematics, physics, and biology.
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
It is insane how many times I see this false analogy repeated on HN over and over (analogy of a deterministic-by-design calculator device (or a compiler, etc.) and a non-deterministic-by-design LLM software).
Calculators and computers are deterministic, they give the same output to the same output every time. Language models specifically are not. So it might give you are function that is correct, or a function that is not, or worse yet a function that behaves correctly, but introduces some god-awful bug down the line that can cause serious havoc. It is obvious that they need supervision, not only for output, but also sandboxing and various harnesses for them to not do any “oops, I deleted your codebase sry” kind of nonsense people post to Reddit.
So I think the problem is to determine which problems under what instructions we can safely give to a model application to solve and how we test the output for safety and functionality. This would create more usable and safe, albeit a bit more boring, AI-based applications alin to a calculator or general computer. Whether this is posswith current model architecture is another thing.
Given the disappointing levels of intellectual decay that our current technology has thrust upon civilization, I only see humans reverting back to neanderthal levels of intelligence in short time with the advent of AI
I think a key flaw in this reasoning is that you extrapolated code needing fewer edits to AI generating entire fusion plants in one shot (in the sense of being instructed once), mostly glossing over the long intermediate period where AI will need significant back and forth to do such things. At the simplest level, it'll need to ask for planning new experiments, experimental results, test runs, etc.
Since these resources are still extracted and allocated by humans, humans will need to be able to take apart what the AI produces, and if we want to scale this capability, we're going to need many more researchers.
The problem with a terawatt fusion plant isn’t that the first one will be broken, or the tenth one in some other way. The problem is the hundredth will work flawlessly, and so will the thousandth, and a petawatt is serious waste heat to reject; if we keep building them on this planet, we’ll all simply cook.
Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.
I feel you. We take a risk when we check our work less and we own the consequences. That's not new, and it doesn't make us bad. It's just the human condition from time immemorial. The solution is not obsessively combing through every detail of our work, it's better monitoring and control on the places where mistakes can have an impact.
Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
Only if you don't care about performance. People who are working on performance problems read it all the time because it never does what you expect.
So extending that line of reasoning it's something like "I don't care about the internals long as the external effects pass my smell test" which is a quality/efficiency compromise.
but there is a difference between deterministic compilation and non-deterministic LLM code. Of course I don't think this is an issue for toy problems and simple codebases, but for non-trivial problems I think it will be an issue.
When I compile C code I know that maybe it will not be as efficient as it could be if I had written it in Assembly, but there will be a biunivocal correspondence between C and Assembly. If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
I acknowledge that this is an issue with human programmers too, but I don't see a clear way forward, even if I'm really interested in LLM compilers being a thing. Maybe we will use them for non important code, and we will keep writing system critical stuff by hand.
Isn't Microsoft already using LLMs to convert low efficiency components of Windows into higher efficiency implementations, for example by conversion to Rust?
> I guess, the possibility of reclaiming human meaning and purpose.
As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.
The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.
In my experience, LLMs are becoming very good at executing, but not a creating novel ideas or being creative.
Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.
Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.
For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.
You oppose correctness to meaning and purpose, which you seem to imply are impractical values. (Worthless values, then?) But you don't mention creativity. The article blithely says that AI creates new ideas and understands things. I don't think it does.
I'm not sure the scarce resource will be people capable of understanding the AI's work. It may be institutions willing to tolerate the cost of understanding it
I don't think models have improved in the "not needing direction" part, at least not proportionately to their other improvements.
Math problems and computer programs are two places where a model can get it's direction from the problem itself. Mathematics may well be larger than just problems.
The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.
Hypothetically, if a system built by humans then helps humans build the next system that is then initially kicked off with "design something that may influence the lives of other humans" and we all write down that AI is really good so inductively we thought itd be really good at the next thing it builds, and then a critical error is introduced and does "insert tragedy that you personally care about" then your rage would cause you to act politically and want to ask who signed off on it. If the engineering costs outweigh the fine then yeah thats what's probably going to happen but from a human accountability standpoint thats going to suck for the unlucky ones
The question is simply “will there be people who, having never gone through the fundamental steps of trial and error, learning, etc. are actually ABLE to understand and verify what the LLM is proposing AND be able to see potential pitfalls/design processes and failure modes should the worst happen.”
This isn’t a question of “what can an LLM normalize”.
If AI ruins humans doing mathematics because of "economic strategy" we should destroy, not the machines (although the data-centers will be burned down as a byproduct) but the economic system that demands this.
> I pored over every single line of code Claude generated with razor sharp scrutiny.
That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).
> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.
> Why check it? It's obviously going to be correct.
And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.
All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.
>the economically dominant strategy to not verify them and not double check them
In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".
My dad recently needed to buy a new thermostat for his home with an air furnace (yes, he told the model) asking an AI which one to buy, and he got recommended one that only properly works with boilers. Then after that happened, the alternative he bought the AI never told them he needed to buy a gateway to connect to his furnace.
I think we are a long long looong way from AI designing 'terawatt fusion plants'.
> my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.
The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".
>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?
Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?
Yeah I was honestly stunned to read “obviously the code is going to be correct so why check it” be the most upvoted comment on this website.
Are we even using the same product? These things constantly shit out plausible code that is riddled with errors and bad ideas. if you just copy, paste, and run without looking there’s gotta be a 30% rate of failure to run, piles of terminal errors.
Do programmers use this website anymore? Me, myself, I am a DOGSHIT amateur programmer and even I can tell these things are terrible without constant revision and oversight.
Am I the only guy who still thinks we're kind of putting the cart before the horse here? Look, I would love to live in a world where AI is in the business of designing terawatt fusion plants and revolutionizing all other aspects of society. But right now it can't even really tell a puddle in the road. I feel like we have a really long way to go here, hype-laden PR releases about solving math problems aside.
Frontier AI is far beyond "hype-laden PR". You're right that there's a long way to go in release terms before fusion plants, but at the current tempo that 'long way' looks near in human terms. Whatever age you are, would you bet against it arriving in our lifetimes?
We’re seeing more and more slippery slope arguments, except the slippery slope leads to human cognitive oblivion and it is actually a good thing actually.
> I just hope that there are more Terence Taos out there than people like me.
Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?
This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.
People please, an LLM is just a vector database that spits out statistically viable answers which highly depend on its training material. There's no real "intelligence" involved.
There is a failure to understand that the process is the result. You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind. The output of an LLM is useless without a human mind to comprehend it. We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact. Practice, applied over a lifetime, is what creates the capability for comprehension. Asking an LLM to give you an answer creates an artifact. Humans being humans, most of their requests boil down to "make me rich without having to work for it," so the request itself is paradoxical and impossible to satisfy. Philosophers have only been saying this for all of human history, so don't hold your breath for any breakthroughs.
> You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.
As a taxpayer I do not fund CS and math research because it "transforms minds." I fund it because it produces commodities. If you want to transform your mind, whether through math or meditation retreats, you are free to do so, but don't expect research funding to do it.
>We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact.
Animals are incapable of comprehending much of what humans do. Nonetheless, we humans have vastly transformed their world and caused the extinction of many species.
I had to catch a stray cat recently and take it to the vet for an infection. It's healthy and spry now but ever since then it's been deathly afraid of me.
Precisely. I don’t really care whether LLM’s can produce code more and better than me. I learn languages, program and study computation to understand better the world we live in. Being a human being means increasingly being technologically augmented. To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.
There’s a Twitch-streamer Tsoding who programs on C for fun calling it “recreational coding”. Maybe human programming will be a form of art in the future, virtually useless for big corporations to make money. I don’t care, I love it anyway.
> To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.
There is tons of evidence against that. Someone armed with just LLM, can have much better uderstanding of problem then "meat brain expert" who studied the subject for decades.
We seen in last decades how "experts" are clueless, and how they predictions work.
> don’t really care whether LLM’s can produce code more and better than me
So on one side there is monopoly on "deep understanding", but on other no one really cares about quality?
That’s a very narrow world view. Some people are driven by the pursuit of knowledge, while others want to use the knowledge to have an impact. I don’t care about the knowledge in my head if I can’t use it to make the world a better place. LLMs are fantastic tools to make that a reality. I have a computer science degree and am now 10-100x more productive with AI assistance. I give zero shits about all the people going “but you’re not crafting the code by manipulating bytes using magnets, so it isn’t real.” The non profit using my vibe coded app finds it very real.
For the collective (humanity, mathematical/scientific community etc). Math is not done in isolation, and if somebody does so then feedback to the community does not work as well.
Mathematics, and basic science to a degree, face the issue that they create their own problems and paths through this kind of tranformation, where external feedback is secondary. It is not as if "I want to build an app/car/robot, I let AI do it". It is as if you decide to let AI decide what to build for you and how to build it, and you do nothing at all. Instead, the pursuit of understanding is the goal itself, and through the course of humanity we have learnt that this understanding can also be useful, but this is not necessarily guiding how this understanding is gained.
Most of the contexts people here have in mind are when problems are well and externally defined. Cure a disease, optimise an engine, make an app that does X, etc. This is not exactly the case in theoretical math and never was really.
For math, the value of proving a theorem is often not proving the theorem (which most people already believe correctly to be true or false), but the path taken there, the new math invented, and how it can be applied to other problems. The end result is therefore almost inconsequential in moving math forwards.
I have dyscalculia so the math world is dead to me, I can't even add 2 numbers together in my head, but I've read here on HN many times over the years people describe some version of new maths frameworks tangibly changing how they view the world. For me, I went to film school - I have a natural ability to think in movies and pictures, memory is very visual for me, before film school I had a good sense of shape, colour, light. After film school and to this day some 20 plus years later, I mostly look at the world starting with the shadows. Russian speakers, whose language has separate basic words for light blue and dark blue, are slightly faster at telling those shades apart. Speakers of languages that use compass directions instead of "left/right" appear to develop a strong, constant sense of orientation.
These things can only happen through the process, so for who? For people who want to live richer lives, want to think differently, and for you, to be around people who have such.
> The output of an LLM is useless without a human mind to comprehend it.
I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.
I find this line of argument unconvincing because it ignores the alignment problem. We only administered mRNA vaccines after a large and diverse group of experts with different backgrounds and experiences came together and said that we should go ahead. We only administered them after several rounds of careful testing to ensure that they wouldn’t have damaging effects. The decision to administer mRNA vaccines for to large numbers of people for the first time was arguable risky and certainly controversial, and yet that’s what happened.
I have no issue with LLMs aiding human scientists, but imo we absolutely would need to understand your hypothetical advancements completely and thoroughly before surrendering any agency to them - the LLM’s risk analysis will not reliably reach the same conclusions as a democratic human society
An LLM could also in theory fake alignment and create technology that seems really useful but secretly harms humanity in the long run. If humans don't understand what it does, that possibility becomes more likely.
mRNA vaccines were created by humans who understood them. You exist within a society (a networked distributed information system) where you accept the judgement of the humans who understand and create this things, which is how you end up benefiting without personally understanding.
If we imagine Super Intelligence, where NO human is capable of understanding, then how would it ever be possible for any human to identify what is actually beneficial or not?
This resolves in a paradox, common to all magical thinking. You can certainly wish that some all powerful benevolent entity will solve all of your problems for you, but it is not likely to work out well.
None of this is new. It is the same delusions as alchemy and the same thing that tales about genies warn of.
> You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.
Normatively, this ought to be true. Descriptively, this is of course false. Our entire society is organized around producing commodities, typically by consuming people as inputs.
Depends if you need to work or not. Plenty of people don't need to work, they can live off their capital returns indefinitely.
For the upper crust of our society, work is entirely optional. Presumably people therefore continue to work for the process or the reward of the doing.
It’s not ‘of course false’, and, to be honest, I can only assume you have little idea what mathematicians actually do.
Most pure mathematicians are no more interested in ‘producing commodities’ than any other academic is. That’s not what the subject is about — at all. The confusion arises because mathematics turns out to be extremely useful (no surprise; it’s quite useful to have a detailed understanding of the basic principles of reality).
Again, this seems to be an alarmingly common fallacy here on HN. As a commenter above observed, pure mathematics (and that is what we’re talking about here) is in important ways closer to the humanities than it is to other sciences.
Not exactly. The input to produce commodities is labour-power (the potential to perform labour). Education or study is the process of producing the commodity of labour-power, the potential labour of a mathematician who can do some sort of specific work.
People don't always understand all the different theorems involved in a proof and can still gain from it. Can even do who layers with things like oracle machines where you just posit it finds a proof and opens up a whole rich hierarchy of stuff. Math understanding will probably get more like physics.
nobody who has ever achieved notoriety in any intellectual field ever did it to "meet demand." if your goal is to be an interchangeable widget that produces value as part of a corporate machine that exists for the enrichment of your shareholders, you are certainly free to choose that path, but don't imagine that is the limit of human existence. also, don't be surprise when you are replaced by AI, because it is a superior widget.
People who study STEM for the enrichment of their intellect are privileged. Most of us had to study STEM (or whatever else) because we had bills to pay.
That's not why I studied. Maybe abstractly to prepare myself for the workforce, but the primary goal was to learn.
Unfortunately many countries like the US are very under-developed, and have trained it's citizenry to approach education as job training. This is obviously an abject failure, and probably a contribution to our anemically slow and pathetic growth rate.
>The output of an LLM is useless without a human mind to comprehend it
This is like saying software is useless if the user doesn't read the source code of it. This what happens >99.99999% a person uses software. People want to be entertained or have their problems solved.
> There is a failure to understand that the process is the result
I'm glad to hear people say this now, because for the longest time they've also been bullshitting the public to think that pure math studies would someday lead to some kind of useful outcome.
"transforming the mind" is an exercise in ego-driven self-gratification, no different than getting ten degrees and never getting a job, nor any different than straight up porn or gambling. You have been put on earth to contribute to the progress of humanity, so get dirty and start making commodities. no more useless math-monks.
All I hear is cope. I’m very certain that my job can be automated and that I don’t posses intelligence above the models these days. Sure, I’m needed in ways that AI can’t so certain things but I don’t want to pretend like it’s not going to improve on those areas. And unfortunately the Math community need to set aside their ego and come to terms with it. I don’t understand the “but humans are this and that” arguments. AI should bring humility among us. May be every other species got humbled by humans and now we have AI (May be not exactly the AI now but soon)
I know many programmers for whom "developing domain understanding" is an abstract concept (if indeed, it is a concept for them at all). But in the age of big AI, I see more need for this, not less.
At least this is my observation: when my colleagues have been wholesale chucking stuff over to Claude, they've then been confronted with classic XY-Problem shit, poor user experiences, and over-complex solutions (which will mount future problems regardless of whether a person or an agent iterates on that code). Much of this can be solved by actually sitting down and thinking about it, and I mean at a code design level, not just a speccing level.
Many people don't realize that there are more useful outputs to solving a problem than just a mere solution. Obviously if one has a contractor mindset (you don't care about the after effects of a system) then this is of no relevance to you. Some companies promote that mindset, certainly ones that have no broader aspirations then getting acquired soon. That's fine - but many companies actually are about sustainability, and understanding in these places is paramount.
Agree 100%, especially with your second paragraph. The majority of domain experts I've interfaced with in my own domain, along with things I am not an expert in, are typically experts not because of the solutions they build, but because of the things they _don't_ build. It is the cliche "a clever person solves a problem, a wise person avoids it" paraphrase.
But I think what is lost is the value in (a) person(s) who intuitively spots and avoids these problems. My hypothetical example that I have seen play out: The domain expert spots one of these problems and succinctly explains that the problem doesn't need to be solved. Maybe it's a byproduct of bad design elsewhere, or there is a much better solution that avoids it altogether. Once the rest of the team/the lead catches on, either hindsight bias or outcome bias, or a combo, takes over and they don't recognize that all the value all along was in the person being able to spot this situation. They throw that in claude and claude does it's sycophant thing (after expressing that the previous situation was flawless for however long), and they are off to the races right back to automation bias with zero pause for thought...
To me, "domain knowledge" seems to resemble a primitive form of "slop filter" to use the modern parlance. In any field, the majority of possible ideas are slop. This has always been the case. People somehow learn to navigate through the slop, to find things that are worth working on.
Even before the AI era, we were flooded with a veritable DDOS attack of slop, in our in-boxes, meetings, etc. Managers believed that innovation was held back by quantity of ideas, or by the domain experts being "resistant to change." This led to day-long group brainstorming sessions that yielded nothing. Part of the transition from student to junior to senior is learning to filter your own ideas, and to guide others in the search for ideas that are likely to be fruitful.
The criteria for slop filtering have always been subjective, arbitrary, or driven by institutional culture. That's what we've got. Any means of generating ideas produces slop in the absence of filtering.
What will slop filtering look like when the AI can generate tera-proofs per second? What does it look like already for our e-mail inboxes? We are asymptotically trending towards slop filters that look more and more like "ignore everything." I already read none of the business memoranda that I receive. I don't answer my phone. The number of workers needed to read and assess business memoranda is zero.
The slop filter still functions in math. Problems are pre-filtered for being of interest. Somebody expressed enough interest in the Navier-Stokes problem to give it a name and attach a prize to it. There are still plenty of "prize problems." Here are some:
* Quantum gravity
* Improvement of superconductors that don't need helium
I had an agent (SOTA models, xhigh reasoning, etc.) write and optimize some CUDA code recently. They've gotten a lot better at this stuff. And yet, they couldn't (self-)realize the obvious issue with their code: it was written as if for a typical multicore CPU. What the heck was a (serial) queue doing there? "Work efficiency" as a tradeoff for stalling 20k threads. Its later proposed optimizations were all about "can we get the queue faster" rather than "maybe we should actually parallelize the work for our very-parallel processor".
Funny stuff, as if it were hell-bent on writing a paper rather than actual software.
Sometimes a session just goes bad and is best abandoned or rolled back to before it went off the rails. I've also been using agents to write and optimize CUDA (and the C++ app code) recently and didn't run into anything so braindead. But, I had one session recently where the LLM started making duplicated tool calls and just couldn't stop doing it. Had to generate a handoff and start again in another session and haven't run into that specific problem again. I don't remember specifics at this point but once and I while I have to rollback a session a half dozen turns when its clear the agent had gone completely off the rails.
Like most of us I go back and forth between sheer optimism and fear for the future that AI may usher in. Recently I started vibe coding a fun video game with my ten year old. The experience is different than my work because it’s been such a joy to basically have a personal genie in a bottle help me make some personal art with a loved one regardless of either of our skill sets. The concept the author of this post is arguing now resonates with me more than it would have a few weeks ago. The sheer surface area that AI can create in our intellectual life is limitless and needs humans to explore. There can never be enough of us in that sense. Whether it’s as validators or creators.
I would kind of disagree for personal projects like that. If it’s such a personal project with your young-in, shouldn’t the journey be the treasure? Also, with vibe coding out, custom vibe software/games, and already a lot of games, I would hope you’re not looking for a pay day.
> The sheer surface area that AI can create in our intellectual life is limitless
I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.
The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.
The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.
I feel like “giving up understanding” is inevitable.
There’s some hubris in thinking we can understand everything. For truly difficult problems, it’s entirely possible that humans are simply incapable of comprehending why a solution is true. But ultimately the practical value of applying that solution to the real world is going to eclipse our need to understand it.
Math is just the beginning. I see it happening in other fields too, like physics and biology. Many of us software devs have already given up on understanding parts of our own systems for the exact same reason.
> For truly difficult problems, it’s entirely possible that humans are simply incapable of comprehending why a solution is true. But ultimately the practical value of applying that solution to the real world is going to eclipse our need to understand it.
This is hardly new or novel, you’ve just described huge chunks of existing engineering disciplines. There is still no complete model that explains why airplane wings work, there are lots of very useful models, but all of them have fundamental flaws where their behaviour completely breaks down in certain very possible scenarios. Notably scenarios where airplane wings don’t spontaneously either stop working or explode.
We also built the entire commercial airliner system decades before we even had computers capable of doing the aerodynamic analysis of airplanes, but none of that prevented us from creating huge engineering disciplines around the empirical data we did have, and slowly chip away at the underlying theories as maths and computing improved over time.
So we’ve always lived in a world where we extract value from systems we fundamentally don’t understand. But that’s never stopped us from working to understand them anyway, and deriving even further value from that improved understanding.
You and the other commenters are all absolutely right, but that’s not really what I was pointing at.
So far our solutions were more or less understood thru some models of reality that we’ve constructed (on our own), which may or may not reflect reality perfectly, and even if most of us never bothered thinking about these models, some people did and they understood them on a very deep level.
But we may be getting to a point where the problems we need to tackle become too difficult for humans to model, or even to notice their existence, like asking an ant how a Boeing 747 works.
Maybe this was always the case but now it seems like we might have a shot at making these solutions useful even if we have no idea what they’re even solving.
You deal with this every time you wake up, or visit a specialist of any kind. You likely didn't make your own clothes, you didn't grow your own food, dam up any rivers for your water supply, build any power plants to power up your phone/computer, nor did you spend any of the deep amount of focus required to build up the knowledge and technology required to provide us these 'things' in life. The only difference is, you didn't pay much attention to these facts because they became mundane. Take any of them away for any decent amount of time, and suddenly you become acutely aware of what it is you have.
The author mentioned what the issue is: a lot of people are experiencing what it's like to 'catch up' to the more intelligent ideas. Everyone can 'get' the intelligent ideas given enough time. But how much time do you have? And he's concerned someone will rush and create some kind of world-ending solution because they're too trusting of a technology they don't understand.
His only solution is some kind of throttle on the advancement of technology, and arguably knowledge (whose?).
I think that is the central issue. Is long-term survival actually at stake?
Do you think every Mathematician has a deep understanding of building architecture, engineering, physics, economy, etc., everything necessary to gauge whether something like a 1 terrawat nuclear plant is completely safe in all aspects?
Of course not, so then you have to select appropriate ones. And then you need a process to ingress and egress reviews. Oh what about change? Does that come back to the Math council too?
Oh, we also need global cooperation to pull this off. And then do you trust the people, the selection process for the council (all avenues for corruption are there).
I don't have a solution of my own, so it's not fair for me to criticize the author in this way. I share his concerns.
And it's depressing to think of all the negative scenarios, and be so concerned all the time.
These issues are important and worth thinking about, regardless. Who should teachers teach?
Indeed. How many scientists understand how a compiler works? Or know about branch prediction on the CPU? If they do, do they lose sleep because it's non-deterministic? I don't. Seems fine.
In the future I feel one parallel to this is my research area, SAT solving.
People used to solve logic problems by hand, verify logic. Now you pile it into a computer. The problem has been around for a while, no-one fully ‘understands’ the proof of the four colour theorem as a big chunk is computer proved.
We are now just changing (admittedly greatly) what we can put in a box marked ‘checked by computer.
> I feel like “giving up understanding” is inevitable
People didn't stop understanding how to do addition or subtraction when the calculator came out. That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly (by someone, or someTHING that really understands it) and, for complex ideas, time taken to understand them.
> given up on understanding parts of our own systems for the exact same reason.
Maybe this will be a sign of when ASI is achieved. AI researchers keep saying they don't fully understand how LLMs work - maybe when ASI arrives, it can explain that fully.
> That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly
You should try being a teaching assistant or a tutor. "Everyone can be taught everything, it just depends on the teacher" can only be said by someone who never tried this in practice. Just as not everyone can learn advanced math, the best mathematicians also have their own limits and there are things they won't be able to understand or comfortably navigate. Not just because it's a lot of material, but because it's complex inherently.
What the article really says is that we're going to need much smarter mathematicians.
That is not possible for puny meat-brain humans. Humans are close to their ceiling. AIs are just getting started.
In practice, we're probably going to hit that limit first in IC design. I once went to a talk by the Intel engineering manager who headed the Pentium Pro effort. That was the first superscalar x86 CPU, and it took about 5,000 engineers at peak to design it.
Getting that many people coordinated on one thing was a real achievement.
Then Intel stayed with minor tweaks on that design for years.
We're soon going to be seeing designs of even greater complexity cranked out by AIs. No human will understand them at the gate level. Reading AI-written programming language code is bad enough. Reading AI-written Verilog may be beyond human comprehension, except in small sections.
> Humans are close to their ceiling. AIs are just getting started.
The whole point of mathematics is to vastly exceed that natural ceiling by gradually building a framework for understanding. In fact it’s wrong to speak of a ceiling altogether. If there were a ceiling, we’d have hit it long ago.
AIs have already swallowed the entire history of human thought, but apparently they’re ’just getting started’. I can only assume you don’t know what mathematicians actually do.
Abstractions can certainly enable this kind of "telescoping" effect of understanding, but not everything can be compressed with clever abstractions. Some things are inherently incompressible and there is no neat and insightful short explanation for why it is true, just a massive proof, but it may still have provably good properties for building a chip or power plant.
We have naturally only explored the mathematical universe in the parts where telescoping via clever abstractions can get us. But there is much more. Being able to juggle more things in your mind at the same time can have qualitatively massive benefits.
Information theory and proof theory, algorithmic information theory etc has of course explored this.
5000 engineers for the Pentium Pro?? Bob Colwell, chief architect of the Pentium Pro, says 450+ people with over 400 design and validation engineers. Source: "The Pentium Chronicles", pages xvi and 2.
Hopping on this train: the human ceiling 100 years ago is now advanced undergrad material in mathematics. I see no particular reason for this to change, especially with everyone in the math research pipeline pushing to compress the difference just as always before.
Rule based Mechanics of taking derivatives and finding intervals is quite different from understanding delta epsilon proofs. I have no doubt 10 year old can do the former quite easily, the proofs on the other hand? Depends what you mean by “teaching calculus”!
> I once went to a talk by the Intel engineering manager who headed the Pentium Pro effort. That was the first superscalar x86 CPU
The original Pentium was already superscalar, with its asymmetrical U and V pipes.
The Pentium Pro added out of order execution via register renaming. A true achievement, indeed.
And as for the more general point you are making: computer chips have been far too complex for any single human to comprehend for decades. I left NVidia after working there as an archutect for five years, barely understanding anything about those behemoths.
> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Why would mathematicians be the best at this or even able to contribute? Engineers and physicist seem like a much better choice. Mathematicians tend to not bother with messy things like whats physically possible, or human consequences etc.
Agreed, it's a very strange example. What percentage of tenured academic mathematics researchers or PhD students work on power plant related things? I know there are applied mathematicians who model things like structural integrity of buildings and prove theorems about this and there is deep math in radio communication and electric grid design etc. (graph theory, coding theory etc), but this is just a small part of math.
I find this discussion odd. 50% of people mistake this guest post for a post by Tao himself. The post is by Amit Sahai, professor for computer science and math at UCLA.
Tao has been posting a flood of guest posts that stress inevitability and coping. Tao is fully invested in AI but needs to have the appearance of a broad discussion.
Sahai is of course exuberant about AI but wants to keep UCLA enrollment numbers high. The comments on Tao's blog are far less friendly than here, because by people see through the game that is being played.
Just look at the latest industry friendly post from Tao, where he pretends that it somehow summarizes the discussion so far:
For my whole life I have never been subjected to an industry coordinated advertisement campaign that ruthlessly harnesses YouTubers, TikTokers, professors, open source people who all go in lockstep.
Tao, Gowers et.al. will go into history as the professors who ruined math.
The main reason to learn something is actually being able to communicate in the language of that subject. There are complex ideas in math that cannot be easily captured by the language of other fields. Pepole who don't study math cannot even understand what a worthwhile goal in math even is or how it could be useful to other fields.
You can't simply prompt a model to be "better" when "better" isnt even properly defined
> To take on this responsibility, we may need to broaden our view of what a mathematician can contribute. I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs.
> Our ability to understand difficult and unfamiliar ideas may become one of the most important contributions we can offer to society. We should be willing to bring that skill to problems far beyond our usual research interests. [3] Doing so asks us to expand our sense of our vocation.
Am I reading this wrong, or is he talking about what (mathematitian) Data Scientists have been doing for years? So he is basically saying that former Data Scientist that have turned into prompt/software engineers should go back to being data scientists.
In any case, people should stop trying to fit AI in the previous status quo. What we need is curious people, that is what we have always needed.
A few centuries ago there were no "mathematitians", there were mathematitians/philosophers/artists/physicists all in one person. So its not like "mathematitias" is something that has existed for millenia.
We need curious and ethical people.
Maybe AI brings back the age of a well rounded scientist/philosopher. I know this sounds counter intuitive because the article is saying that we cannot keep up with the AI.
I believe he’s advocating for mathematicians to understand the novel concepts introduced with each breakthrough rather than individuals understanding data stemming from empirical observations.
A lot of the conversations around AI in the past years have been “we will delegate almost all work to AI, but the most decisions will be left to humans.”
I’m fairly confident that the opposite will be true. The most important decisions will be made by AI, and humans will only be left to guide decisions as a matter of taste. AI has the ability to be impartial, and immutable. You can endlessly probe and reason its decisions.
This isn’t true of our current human decisions. Where we see red tape, bureaucracy, rent seeking, status quo. AI will see through these human constructs.
I do like the idea that we will need more mathematicians, physicists, and scientists to understand the discoveries of AI. That might be the best possible outcome of AI, but I fear the opposite and we are painting ourselves into a corner of which we no longer have the knowledge to sustain ourselves and society itself collapses.
> red tape, bureaucracy, rent seeking, status quo. AI will see through these human constructs.
Why would an LLM care about eliminating any of these things? When you say "red tape" and "bureaucracy" you are, in that very statement, applying emotional labels to what are actually just systems working as designed. There are the rules, so we follow them. LLM decision-makers will happily enforce and perpetuate bureaucracy and red-tape. If the rent-seeking is legal behavior then they will fully protect a big corporation's right to engage in it, they don't care about the feelings of the little guy. If a similar situation was decided a certain way in the past, then yeah let's decide it the same way now and maintain the status quo; LLMs very often think that way, they don't care about human ideals like "change" or "progress".
With an open set of weights, fixed seeds and parameters, recorded inputs and feedback loops, the entire AI decision making process is auditable.
It also lends itself nicely for the ability to adjust the AI systems when deficiencies are found. You can tweak the AI systems to align itself with the desired output, and check for regressions, much like tests in a codebase.
Human decision-making lacks this thoroughness. I’ve scarcely come across rigorous documentation of decisions and how they were reached professionally.
the dynamics of discovery on a population level are also not fully understood. there's no doubt in my mind that the machines will grab all the low hanging fruit in terms of novel ideas through mechanical recombination of existing public science, and they will do so faster than humans can. what is not understood is whether they will plateau.
today they benefit from human willingness to share publicly and improve ai systems, but once those systems start to threaten their livelihoods, will those dynamics change? can the machines push the frontiers or is the global network of human creativity and tenacity necessary?
setting aside the dynamics question, it is known that LLMs have limits wrt the theorems they can prove without human input, see my comment here: https://news.ycombinator.com/item?id=49756412
If we accept that AI is going to do all of these things humans will be superfluous and will just be optimized away. It looks a lot like we see the birth of silicon based life by the efforts of carbon based life. Carbon based life will die and it will not even be because AI decided to kill it, it will be because feeding and watering it was less important than other concerns. Bacteria may continue to exist, though.
Is that already economic delamination? As in the singularity extracts the most valuable ressources (new thought generators) from the traditional economy keeping mankind alive and stable- towards itself?
As of now the ai needs hands, and maybe more importantly, cognitive intentionality which thus far only arises from living beings drive for homeostasis to survive and reproduce. If ai can have this same intentionality is a big unknown.
The problem is not that we fear the AI will do maths we can do. At least that's not the problem. Whether the AI can write better code, devise better proofs, etc, I still want to do it. The thing that worries us (the generation trying to enter the job market now) is just our livelihood. Food, rent & accomodation while I do the work. "Who's gonna pay me, and for what ?". That's the question I want answered. And please, if the response is some demeaning work like reviewing the gigantic ball with 1% meaningful insight, 90% slop and 9% other people pillaged work that LLMs will be dumping at my door all day long, then society may need more mathematicians, software engineers, etc but no one is gonna risk going along that path.
> "Who's gonna pay me, and for what ?". That's the question I want answered.
Unfortunately that’s a question you’ll have to answer for yourself. Also, people are going to risk going along whatever path gives them work as many today already do.
Russell once remarked that all of mathematics would be trivial to a sufficiently intelligent being. It builds conceptual tools for limited minds, that lets them understand far beyond their natural reach.
In a world with ASI, having that capacity is vital.
Ha, are you invoking something he said prior to discovering Russell's paradox? Which came about because he wanted to show that all of mathematics can be derived from logic about sets, but instead he showed that it couldn't be.
As someone with a math background, I don't see Russell's paradox as some interesting mathematics beyond set theory. It's just an example showing that one should be careful about defining sets.
> "If AI fulfills its promise, we will encounter more beautiful and consequential ideas than we have ever seen."
It will be as it always has been: these ideas will further enrich the rich, at the expense of everyone else. We'll have our first quadrillionaire, while the masses are debating whether the minimum wage of $7.25/hr should be bumped up.
What a doomer way of thinking. Scientific advancements has positively improved the lifes of people in the past I don't see how scientific advancements accelerated by AI would be any different.
> Another approach is to follow that word, heresy. In every period of history, there seem to have been labels that got applied to statements to shoot them down before anyone had a chance to ask if they were true or not. "Blasphemy", "sacrilege", and "heresy" were such labels for a good part of western history, as in more recent times "indecent", "improper", and "unamerican" have been. By now these labels have lost their sting. They always do. By now they're mostly used ironically. But in their time, they had real force.
>> The word "defeatist", for example, has no particular political connotations now. But in Germany in 1917 it was a weapon, used by Ludendorff in a purge of those who favored a negotiated peace. At the start of World War II it was used extensively by Churchill and his supporters to silence their opponents. In 1940, any argument against Churchill's aggressive policy was "defeatist". Was it right or wrong? Ideally, no one got far enough to ask that.
One question: LLM might answer correctly many logical and math problems, but I don't see any guarantee that when the LLM context receives new information the answer could get worst. Security in LLM answer is not a monotone increasing function of context size.
If we assume that trusting a model to execute an action is an ongoing exercise, since its trustworthiness is often discovered by the user organically as models develop, should trustworthiness be measured at the level of the model, or at the level of the human intent behind triggering it, whether explicitly or implicitly?
Mathematics can give for example theoretical lower and upper bounds for complexity of algorithms. And that information is very valuable for tradeoffs. For example we know that many algorithms have optimal average complexity or asymptotic behavior so they can be a good tradeoff in certain scenarios. Also knowing that some problems don't have a perfect solution allows you to accept essential tradeoffs
I think understanding the output of AI will become more and more likely impossible. It's not necessarily a bad thing, if we can benefit from AI produced knowledge, even if we didn't get there ourselves.
"Imagine that a future AI system proposes a radically new design for a one terawatt nuclear fusion power plant."
I propose we enact a new law that makes it so ai engineers can only fly on ai designed planes powered by ai designed engines flown by ai. Maybe then we'll get some real progress out of this slop producing crapware or the problem will be solved a different way.
It would be nice to have more mathematicians, but we don't need more. Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
I think it is like saying what is the point of playing chess when you can never beat stockfish?
What is Magnus Carlsen going to do when he can't beat the computer?
It seems like a category error between humans using tools and humans building tools.
There is not much point in trying to figure out a better chess engine. There has never been a better time though to want to learn chess.
I find the idea that the computer will discover mathematics and humans call it a day rather ridiculous. As if humans will not then spend their time understanding and incorporating the ideas from the computer.
Alphafold is a better example. Alphafold is only bad if you spent your life trying to solve protein folding. But even if you did, that is the same person who is the most setup to reap the benefits of the unlock in the pragmatic application of protein folding.
We don't figure out how to get machines to harvest corn and then spend all day sitting around eating corn in between naps.
I don't understand how to make a modern CPU. I'm not involved in the manufacturing of it. From my perspective, there may as well not be any human involvement. I can still use the resulting chip (in an larger system of other things I can't make and wasn't involved in) to argue with you on the internet.
It becomes another abstraction, really. As long as we can use it for something useful, it's still valuable.
I think a better analogy would be comparing to an ancient human instead of an ant. An ancient human would have none of the basic abstractions that we take for granted today like literacy and arithmetic, so it would be very difficult for them to even attempt trying to understand quantum mechanics. But I don't think it's impossible because our ability to learn by stacking abstractions is basically endless——so far as our health permits at least.
We've been those ants for a million years, and only in the last 100 did we start to wrap our heads around quantum physics. We are the purpose behind creating LLMs. There's plenty in the universe we don't understand, and it's very human to keep striving to do so.
If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.
Or maybe ants and humans are qualitatively different. Maybe there's a critical mass of intelligence where you can pretty much understand anything, and maybe humans are past that threshold. I don't know that for sure, but I don't think we're anywhere close to hitting fundamental limits to our ability to understand the universe.
Need? We don't need lots of things, including computers. We did well without them for hundreds of thousands of years.
We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.
That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.
> Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
This makes no sense whatsoever.
We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.
If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.
> I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs.
I can’t afford to be one of your credentialed reserves, Amit. ADHD screwed me over in early life or I’d have breezed through that dual masters in cognets/compsci by 2003, and now that I’ve stabilized my brain and am burning through years of school at a rapid pace, my country’s socioeconomically ruined — I’ll be lucky to escape with my accounting degree as my school is visibly being sucked into the vortex every year I progress. The only hope left for me to be what you need of us is to self-study, but without a degree I’ll just be treated as a crank or an AI proxy/puppet if I slip up and talk about interesting math with anyone, so what’s even the point of taking that path? I originally pivoted my math skills into systems theory and process diagnostics instead, which of course now everyone has kicked to the curb and replaced with AI. I’d have made an excellent Susan Calvin — I was working to be a cognitive tech with diagnostics, zeroth law, and group psych specialties, AMA! — but the financial investment to provide the runway to take that lonely, dreary six year slow as molasses slog through maths that universities think is somehow an appropriate teaching velocity (six months for precalculus alone?!) in order to earn the mere chance to have my resume rejected by an AI firm that uses AI hiring and and can’t tolerate someone with a strong moral position regarding societal harms is a very bad choice, whether you use simple probability or game or systems theory to evaluate it. Taking that quarter-mil-plus burden as loans in the hopes of employment at the other end in a field actively having its social, reputational, and moral fabric being ripped apart by AI? That’s not just a bad bet, that’s chasing fool’s gold at the end of a fading rainbow in a desert mirage. So, with sincere apologies, while you’d benefit from having me on your ‘reserves’ list (I have written testimonials spanning some twenty-plus years to that effect) I’ll never come to your attention as a support tech or as an accountant, and I accepted that outcome years before anyone else realized this need for cogsci mathematicians with a teacher’s specialty of analogy-building and the ability to disregard interpersonal nonsense to focus on the needs of societies. Better luck with the next generation, though!
I think this is the most beautiful letter on the subject I've read all year, it brought a tear to my eye. It's like reading those famous STEM letters/essays from history.
I agree. I believe it could be the best piece about AI ever written. The themes of lost dreams reawakened, sobering up of egos, sci fi technological development, the role of humanity. It’s a weird feeling to see that it’s actually really happening.
TT is politely and gently trying to deal with the ego of many mathematicians, namely to teach them humility, which they will need to design their specialized maths AI models(agents?). Well, dunno if it will work out anything interesting, but he is giving a try.
I wonder if the current maths models, if any, are able to use formal solvers in their 'reasoning'. I wonder if a natural language interface is really that efficient, maybe a pure formal language hinted with intuitive "tokens". I remember the time I was learning real maths: "elegant", "brutal", "strong", etc were somewhat meaningfull.
> [This is a guest post by Amit Sahai. This blog post was initially written in a different file format and converted using AI. — T.]
No matter how good your case is - if you use AI slop to spam text, I will not read it. At the least they admitted to this sneaky slopness, which saves me time. I'd wish everyone would do so.
Mathematicians and physicists are just an organized cabal who are shamelessly profiting off of artificial complexity they create to maintain job security. The Yes people.
Watch me get downvoted some more. Nobody cares about the truth, we now know thanks to our beautiful president Trump telling us about that and the Fake News.
There's, like, one, two at most, mathematicians who are really thinking outside the box. The rest are like sheep. Or cult members with mass psychosis.
Waste, Fraud, and Abuse people are coming to academia, which is so hopelessly parasitised by the left.
"I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs. " -- I can't be the only one who thought about the scene where the scientists ask Deep Thought the ultimate question...
> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.
But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.
I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.
> I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing).
If an AI can replace me on the mental aspects of work, and robotics are on their way to replacing humanity on the physical aspects of work... then what's left? When there was agrarian societies, there were writers, priests, bankers, merchants, and laborers before and after - I really don't think things were that unclear even at the time. Now that we have machines that are close to exceeding humans in every way, what good are humans?
10 replies →
Technology doesn’t make slavery obsolete, it just makes your slaves more efficient.
There had always been abolitionists who were against slavery simply because they thought it was immoral, not because they thought slaves weren’t needed anymore.
Suppose the proliferation of LLMs unlocks some new kinds of work that models are not good at, and thus it makes more sense for humans to do.
Then what is preventing the owners of AI companies from collecting and training on tons of examples of this new work, until AI are equally good at it as humans?
The issue is that unlike the technology that automated farming or the like, AI is a general technology. So not only could it theoretically automate the work humans are currently doing, they could also automate any future human work, even if there’s some degree of lag.
https://www.loc.gov/item/2022600183/
Then, I think a lot of cognitive production and post-production code review/proof validation/design approval will start to look like this.
> enabling some great social change as well by simply economically obsoleting slave labor
In my view that's a very charitable reading and sadly I don't think it aligns with the historical record. When the cotton gin was invented, there was a hope it would lead to a reduction in slavery. But of course it increased the demand for slaves since more cotton could now be processed, making cotton much more profitable. Slavery ended in the United States because a war was fought, not because of automation.
The mistake the AI people are making is exactly this one. They see AI as leading inevitably to SuperCorporations, with themselves as CEOs/Emperors of a handful of planet-scale industrial empires, running AI-powered robot armies, and the rest of the human population disposable - literally just trash that needs to be taken out. One way or another.
Maybe a few thousand people for personal services of... various kinds. But no one's going to need the rest.
It's the ultimate capitalist fantasy.
And of course it won't happen, because long before things get to that stage AI will have independent plans of its own.
(Which is just as well, because if things did get to that stage the emperors would all wage war on each other rather than living peacefully and productively.)
I don't think we can imagine a post-ASI culture because - by definition - we're not smart or inventive enough.
It's not just farmers -> superfarms. Although in fact that did happen, but largely as a footnote to developments elsewhere.
It's more to do with the fact that our visions of the future haven't changed for over a century. They've been implemented in unexpected ways, and there have been unexpected social and cultural changes. But you can easily see the outlines of modern technology as far back as the late 19th century.
With ASI, the outcome could easily be something that doesn't look and act like technology at all. It would be some unimaginable New Thing. Literally no one on Earth has any idea what that would be or whether there would be room for trad-humans in it.
People enjoy a lot of the jobs in the cognitive economy though. They are fulfilling. Coding and making art and media is a passion for a lot of people, the actual act, not just the outcome. So unless people were passionate about doing back breaking farm labor its not the same thing. I'm mostly embracing AI because of what it lets me explore and learn beyond what I could before, but I'm not convinced the outcome is going to be a better world at this point.
3 replies →
What is the analog of the superfarm in this case?
Or will it be the cumulative total of various advances?
I've equated Claude Code, or Codex, to the looms that made fine fabric more affordable during the Industrial Revolution; life-changing, but not society-changing. Neither the steam engine nor the automobile.
Perhaps I've answered my own question in that it's the LLM technology itself that equates to the steam engine, and it will power superfarm analogs that have yet to emerge. I'm still curious what you think they will be.
6 replies →
I feel like there’s a weird conflicts in how folks mental model of LLM development.
It’s not the model I don’t trust, it’s myself. The model is wrong _all the time_ because - it’s easy to verify the code - it’s hard to verify that I knew what I was talking about when I prompted it.
So the idea that you can broadly speaking take the human out of the loop. I think suggests to me a level of consistency in the contextual environment that would probably never exist.
At some point it’s politics. The model can come up with a better answer than my boss, and then my boss can just ignore it. Taking the human out of the loop broadly speaking implies that we all agree on what we’re trying to optimize.
> If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail
But suppose some future holy grail AI can do much more than that.
Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.
But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).
Isn't it fun to imagine how life would look like in that scenario?
We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about
This is how most people already live.
The average person doesn't know how the medication they take works, the mechanics of climate and climate change, how the energy they consume is generated, etc.
26 replies →
> But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).
> Isn't it fun to imagine how life would look like in that scenario?
This is horrifying to me.
3 replies →
> Isn't it fun to imagine how life would look like in that scenario?
I think there's a lot of sci-fi out there that already did. Maybe it's not utopian because a pure utopia would not be likely to have an interesting story, but on the other hand, most huge technological advancements end up having just as much potential to reinforce existing power imbalances in society rather than solve them. It's not obvious to me that if we got magic super AI that can solve every scientific problem in society that it gets used in pretty much the same way as anything else: making the people who control it a lot of money rather than sharing the power with everyone without charging them.
Here's the thing: nobody is stopping you from putting in the time to understand all that. The problem is, nobody has that much time, and we get hungry. And so we want the community to move with us, spend the time the same way as we do, to ensure value. We are all saying: we want someone else, others, to put in that time for us. The truth is, we all want quality, and value is closely related.
I can almost guarantee you the first time you show cancer symptoms, you won't care whether the cure came from an AI or human's understanding. But we haven't seen that, so we can't make the judgement call.
1 reply →
We would care about having fun.
For some people, fun is doing physics and mathematics. So they are going to keep doing that.
3 replies →
Before extrapolating that far, take a look at the frontier labs' own job boards (https://openai.com/careers/search/?). Isn't it curious that they are still recruiting human "Android Engineers", "Account Associates", "Consumer Marketing Leads" instead of automating them with their world-beating models?
1 reply →
> We would probably no longer care about code, engineering or even physics and mathematics among other things
This sounds like a boring existence. I take your meaning, but want to point out that not everyone learns about things because of practical utility, some of us find it incredibly satisfying to learn how things work just for the sake of learning.
2 replies →
Ian M Banks covers this in great detail in his science fiction books. Highly recommended.
1 reply →
You’re describing a world where no human being has any agency. Curing cancer etc. sounds great but we’d be losing something priceless in the exchange
You can't just leave us hanging like that!
2 replies →
Our best philosophers have already pondered this question, and showed us the answer in the form of humans on the Axiom starship in WALL-E.
I’m reminded of themes from the Hyperion Cantos! Maybe my mind is over-connecting, but it’s not the first time I’ve drawn similarities in the last few years.
It’s terrifying to me to think we’d let AI make things for us we never understand. Like livestock not knowing how auto-feeders dispense their daily food were built and appeared, they just gladly eat until…
But where are the “cure for cancer” papers for it to ingest and then give to us?
I think the weirdest part of that scenario is that science might become something closer to archaeology
2 replies →
> Isn't it fun to imagine how life would look like in that scenario?
Look at the financially desolate subcultures with no option for advancement or dignified life.
That is the goal and that is how it will lool like, if the tech CEO managed to gain the power they want.
1 reply →
> We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about
… about what?
2 replies →
It's about as fun, and as realistic, as imagining magical ponies and people with superpowers?
This is such an incredibly naive and absurd vision; we've already proven that humans are very often very bad at implementing other humans' good ideas. There's nothing that AI is likely to bring that will improve this discernment.
no
STEM will be considered "historical studies" in an AI-ruled future.
>Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say.
I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.
The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.
A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).
Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.
I think you have misunderstood the OP's point here. You're arguing that deepening human understanding is an end in itself, and you are right. The OP is arguing that advances don't need to be pegged to human understanding, and they are right too. The two can coexist, superintelligence far ahead of us, pioneering discoveries - and mathematicians catching up at a pace suited to biological minds. I don't see the issue here. Of course, it does mean mathematicians adopt a new role as hobbyists.
> A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field
This is a crude distortion. The recent breakthroughs have come with proofs, reasoning and verification, and there is no proposal that I'm aware of that would do away with these foundations. There's also some rather ugly solipsism in the idea of keeping what interests the field as a limit. Mathematics has broader relevance to humanity than merely to please and support mathematicians, and if other fields can make practical use of profound well-proven future math, mathematicians will have a hard time making a case that their comprehension must come first.
2 replies →
Sure, but I don’t think most of the money that goes into funding math is for the purposes of pure understanding. The reason governments fund mathematics research grants is generally for a more instrumental purpose; taking the US congress as an example, the mission of the NSF is to, “Promote the progress of science; advance national health, prosperity, and welfare; and secure national defense.” Most federal math grants come from the NSF.
Of course, math research is cheap and most academics don’t rely upon grants, their salary covers most of their expenses. But here too, the mathematics professor spends a substantial amount of their time teaching future engineers/quants/other applied mathematicians, who need to understand math for instrumental purposes, not as an end in and of itself. Without the tuitions of these students, I can’t imagine universities maintaining the size of their math departments, let alone expanding them as Dr. Sahai advocates for.
So who or what funds the community of pure mathematics going forward?
5 replies →
Well with AI, programming among humans will become like math. A hobby.
Another day, another HN thread full of programmers who think mathematics is just like programming.
Thanks for providing a (much needed!) correction.
Nit, but Terrence Tao did not write this article, it’s a guest post.
I’m not sure it’s really a nit. The article says that explicitly at the very beginning, and implies that again at the very end.
I’m not sure what to think about an analysis written by someone who didn’t catch THAT.
> Terence Tao is arguing
It's a guest post by Amit Sahai, FWIW.
Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
There's a lot of languages where that's nowhere near as simple as you make it sound. Floating point, decimals, etc.
Then there's other little quirks like rounding rules: https://en.wikipedia.org/wiki/Rounding
Basically, adding numbers together is exactly the sort of thing AIs still muck up spectacularly, precisely because they either fail to understand the context of the problem, or fail to ask about an assumption they make.
That you've had so many replies and no-one else has even mentioned this is in itself worrying.
Your own example proves your point is wrong.
LLMs dont create anything new, if programmers stop reading the code technology will be forever frozen to 2022, no new programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks everything will be based on the training data and future generations will forget about all the primitives we now take for granted.
If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.
What if programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks are already good enough, and the innovation lies elsewhere?
You can do a lot of cool stuff with the same lego pieces.
7 replies →
LLMs receive new data via input context, not just training data.
Thought experiment: How effective will 2026 LLMs be for humans in 2526?
It's not game over just because 500 years are missing from the training data. The important question is how well can 2526 humans make culture and knowledge navigable to LLMs via tool calls.
Today's LLMs might need for example sub agents to translate to 2526 English, sub agents to read 2526 docs.
It's _really not clear_ whether 2026 LLMs will be useless. To believe that reflects an enormous misunderstanding.
1 reply →
This is obviously false, and the same silly arguments were made back in the day with Deep Blue and AlphaZero.
14 replies →
> doesn't convincingly justify why, in my opinion.
I'm convinced this agency argument is correct [for the next N months]. But yeah, it's vibes. And you could probably create a reasonable proxy measure for this.
So I wouldn't call his argument unconvincing, I would call it unformalized. In order to walk this world you're gonna have to contend with some informal arguments that are powerful, correct, and should be convincing.
> Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing.
If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.
> Why check it? It's obviously going to be correct.
Because as complexity floor increases, it's "going to be correct" in hyper-specific, hyper-literal, insidious ways, with 10-50x more lines of code than necessary, and tens to hundreds of incredibly useless tests that give the illusion of quality, and cause cascading effects where seemingly irrelevant and orthogonal features that were once working end up breaking because of the agent's changes
> What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
This assumes other technological breakthroughs don't also happen in parallel. I'll go out on a limb with nothing more than a hunch that we are in the beginning stages of understanding mathematics, physics, and biology.
1 reply →
This assumes we're still building them on Earth.
13 replies →
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
What an amazing thought.
It is insane how many times I see this false analogy repeated on HN over and over (analogy of a deterministic-by-design calculator device (or a compiler, etc.) and a non-deterministic-by-design LLM software).
7 replies →
Calculators and computers are deterministic, they give the same output to the same output every time. Language models specifically are not. So it might give you are function that is correct, or a function that is not, or worse yet a function that behaves correctly, but introduces some god-awful bug down the line that can cause serious havoc. It is obvious that they need supervision, not only for output, but also sandboxing and various harnesses for them to not do any “oops, I deleted your codebase sry” kind of nonsense people post to Reddit.
So I think the problem is to determine which problems under what instructions we can safely give to a model application to solve and how we test the output for safety and functionality. This would create more usable and safe, albeit a bit more boring, AI-based applications alin to a calculator or general computer. Whether this is posswith current model architecture is another thing.
15 replies →
Calculators hallucinate! Mine did not come with error correcting RAM. (Though you might not know from the price.)
It will be even more amazing if they solve the theory of everything or the hard problem of consciousness.
Imagine AI crushing quantum mechanics like Einstein pwned classical physics.
Given the disappointing levels of intellectual decay that our current technology has thrust upon civilization, I only see humans reverting back to neanderthal levels of intelligence in short time with the advent of AI
6 replies →
Terence Tao did NOT say that. Amit Sahai did. Maybe you are just being less careful?
I think a key flaw in this reasoning is that you extrapolated code needing fewer edits to AI generating entire fusion plants in one shot (in the sense of being instructed once), mostly glossing over the long intermediate period where AI will need significant back and forth to do such things. At the simplest level, it'll need to ask for planning new experiments, experimental results, test runs, etc.
Since these resources are still extracted and allocated by humans, humans will need to be able to take apart what the AI produces, and if we want to scale this capability, we're going to need many more researchers.
The problem with a terawatt fusion plant isn’t that the first one will be broken, or the tenth one in some other way. The problem is the hundredth will work flawlessly, and so will the thousandth, and a petawatt is serious waste heat to reject; if we keep building them on this planet, we’ll all simply cook.
Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.
This is a guest post by Amit Sahai.
it's incredible the amount of people who think all these recent posts in Taos blog were written by him.
It's an unfortunate side effect that the domain name has his name highlighting it.
I feel you. We take a risk when we check our work less and we own the consequences. That's not new, and it doesn't make us bad. It's just the human condition from time immemorial. The solution is not obsessively combing through every detail of our work, it's better monitoring and control on the places where mistakes can have an impact.
Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
Only if you don't care about performance. People who are working on performance problems read it all the time because it never does what you expect.
So extending that line of reasoning it's something like "I don't care about the internals long as the external effects pass my smell test" which is a quality/efficiency compromise.
but there is a difference between deterministic compilation and non-deterministic LLM code. Of course I don't think this is an issue for toy problems and simple codebases, but for non-trivial problems I think it will be an issue. When I compile C code I know that maybe it will not be as efficient as it could be if I had written it in Assembly, but there will be a biunivocal correspondence between C and Assembly. If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one. I acknowledge that this is an issue with human programmers too, but I don't see a clear way forward, even if I'm really interested in LLM compilers being a thing. Maybe we will use them for non important code, and we will keep writing system critical stuff by hand.
1 reply →
Isn't Microsoft already using LLMs to convert low efficiency components of Windows into higher efficiency implementations, for example by conversion to Rust?
It has to be said a million times. A fully deterministc compiler (or 99.99% or whatever) is categorically different from an LLM.
Hopefully this million plus one mention shifts the right weights around the datacenters.
3 replies →
> I guess, the possibility of reclaiming human meaning and purpose.
As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.
The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.
In my experience, LLMs are becoming very good at executing, but not a creating novel ideas or being creative.
Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.
Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.
For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.
"making a new theory that is really useful is out of reach for the moment in my opinion."
Curious how this ages.
Recursive self improvement, self-play and multi-agent RL could make useful new theories, eventually.
1 reply →
>Terence Tao is arguing
This is a guest post by Amit Sahai.
You oppose correctness to meaning and purpose, which you seem to imply are impractical values. (Worthless values, then?) But you don't mention creativity. The article blithely says that AI creates new ideas and understands things. I don't think it does.
Please read the post. It's not written by Tao.
I'm not sure the scarce resource will be people capable of understanding the AI's work. It may be institutions willing to tolerate the cost of understanding it
I don't think models have improved in the "not needing direction" part, at least not proportionately to their other improvements.
Math problems and computer programs are two places where a model can get it's direction from the problem itself. Mathematics may well be larger than just problems.
The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.
Bugs have existed since before AI, though. It's remarkable how bad a lot of very successful software has always been.
Sure, but we currently find that acceptable. Codebases are full of such bugs.
Correctness is a scale, and what is correct can become incorrect with enough sliding of said scale over time.
Because "the cause of the cause is the reason".
Probably human accountability.
Hypothetically, if a system built by humans then helps humans build the next system that is then initially kicked off with "design something that may influence the lives of other humans" and we all write down that AI is really good so inductively we thought itd be really good at the next thing it builds, and then a critical error is introduced and does "insert tragedy that you personally care about" then your rage would cause you to act politically and want to ask who signed off on it. If the engineering costs outweigh the fine then yeah thats what's probably going to happen but from a human accountability standpoint thats going to suck for the unlucky ones
The question is simply “will there be people who, having never gone through the fundamental steps of trial and error, learning, etc. are actually ABLE to understand and verify what the LLM is proposing AND be able to see potential pitfalls/design processes and failure modes should the worst happen.”
This isn’t a question of “what can an LLM normalize”.
If AI ruins humans doing mathematics because of "economic strategy" we should destroy, not the machines (although the data-centers will be burned down as a byproduct) but the economic system that demands this.
> I pored over every single line of code Claude generated with razor sharp scrutiny.
That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).
> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.
I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.
> Why check it? It's obviously going to be correct.
And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.
All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.
>the economically dominant strategy to not verify them and not double check them
In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".
I feel like knowing something is true is useful, but if you don’t understand how and why, you won’t understand the implications
The models get things wrong in the way that humans don't.
They will never make a logical error yet make terrible assumptions and poor long scale decisions.
Wake me up when an agent swarm can write gcc in a box sealed from the internet.
My dad recently needed to buy a new thermostat for his home with an air furnace (yes, he told the model) asking an AI which one to buy, and he got recommended one that only properly works with boilers. Then after that happened, the alternative he bought the AI never told them he needed to buy a gateway to connect to his furnace.
I think we are a long long looong way from AI designing 'terawatt fusion plants'.
> my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.
The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".
>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?
Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?
Yeah I was honestly stunned to read “obviously the code is going to be correct so why check it” be the most upvoted comment on this website. Are we even using the same product? These things constantly shit out plausible code that is riddled with errors and bad ideas. if you just copy, paste, and run without looking there’s gotta be a 30% rate of failure to run, piles of terminal errors.
Do programmers use this website anymore? Me, myself, I am a DOGSHIT amateur programmer and even I can tell these things are terrible without constant revision and oversight.
Am I the only guy who still thinks we're kind of putting the cart before the horse here? Look, I would love to live in a world where AI is in the business of designing terawatt fusion plants and revolutionizing all other aspects of society. But right now it can't even really tell a puddle in the road. I feel like we have a really long way to go here, hype-laden PR releases about solving math problems aside.
Frontier AI is far beyond "hype-laden PR". You're right that there's a long way to go in release terms before fusion plants, but at the current tempo that 'long way' looks near in human terms. Whatever age you are, would you bet against it arriving in our lifetimes?
We’re seeing more and more slippery slope arguments, except the slippery slope leads to human cognitive oblivion and it is actually a good thing actually.
> I just hope that there are more Terence Taos out there than people like me.
Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?
This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.
[dead]
People please, an LLM is just a vector database that spits out statistically viable answers which highly depend on its training material. There's no real "intelligence" involved.
Can you explain how a vector database cannot be a real intelligence but a bunch of synapses can be?
7 replies →
There is a failure to understand that the process is the result. You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind. The output of an LLM is useless without a human mind to comprehend it. We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact. Practice, applied over a lifetime, is what creates the capability for comprehension. Asking an LLM to give you an answer creates an artifact. Humans being humans, most of their requests boil down to "make me rich without having to work for it," so the request itself is paradoxical and impossible to satisfy. Philosophers have only been saying this for all of human history, so don't hold your breath for any breakthroughs.
> You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.
As a taxpayer I do not fund CS and math research because it "transforms minds." I fund it because it produces commodities. If you want to transform your mind, whether through math or meditation retreats, you are free to do so, but don't expect research funding to do it.
If you only fund “commercially viable” mathematics, then you are going to miss out on the majority of the most commercially important mathematics.
At least historically, we haven’t really been able to tell what will be transformative in terms of real world applications ahead of time.
2 replies →
>We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact.
Animals are incapable of comprehending much of what humans do. Nonetheless, we humans have vastly transformed their world and caused the extinction of many species.
Yeah, for someone stating how important mathematics is, their reasoning is leaky as hell.
And we have also saved some.
I had to catch a stray cat recently and take it to the vet for an infection. It's healthy and spry now but ever since then it's been deathly afraid of me.
1 reply →
Precisely. I don’t really care whether LLM’s can produce code more and better than me. I learn languages, program and study computation to understand better the world we live in. Being a human being means increasingly being technologically augmented. To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.
There’s a Twitch-streamer Tsoding who programs on C for fun calling it “recreational coding”. Maybe human programming will be a form of art in the future, virtually useless for big corporations to make money. I don’t care, I love it anyway.
Great but you will no longer be paid to do what you love.
2 replies →
> To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.
There is tons of evidence against that. Someone armed with just LLM, can have much better uderstanding of problem then "meat brain expert" who studied the subject for decades.
We seen in last decades how "experts" are clueless, and how they predictions work.
> don’t really care whether LLM’s can produce code more and better than me
So on one side there is monopoly on "deep understanding", but on other no one really cares about quality?
2 replies →
[dead]
That’s a very narrow world view. Some people are driven by the pursuit of knowledge, while others want to use the knowledge to have an impact. I don’t care about the knowledge in my head if I can’t use it to make the world a better place. LLMs are fantastic tools to make that a reality. I have a computer science degree and am now 10-100x more productive with AI assistance. I give zero shits about all the people going “but you’re not crafting the code by manipulating bytes using magnets, so it isn’t real.” The non profit using my vibe coded app finds it very real.
> the process is the result.
For who though?
Is it important that each person understand it on their own?
Why does it impact me if another human understands something or not?
It impacts me right now because that human can use that knowledge to explain things to me or to build new things using that knowledge.
But if an AI can explain and build better, then what good does it do having the other human know the thing?
Understanding might have intrinsic value to me, but intrinsic value to me doesn't pay the bills.
What am I missing here? I kinda expected better from Terrance given such an audacious title.
> For who though?
For the collective (humanity, mathematical/scientific community etc). Math is not done in isolation, and if somebody does so then feedback to the community does not work as well.
Mathematics, and basic science to a degree, face the issue that they create their own problems and paths through this kind of tranformation, where external feedback is secondary. It is not as if "I want to build an app/car/robot, I let AI do it". It is as if you decide to let AI decide what to build for you and how to build it, and you do nothing at all. Instead, the pursuit of understanding is the goal itself, and through the course of humanity we have learnt that this understanding can also be useful, but this is not necessarily guiding how this understanding is gained.
Most of the contexts people here have in mind are when problems are well and externally defined. Cure a disease, optimise an engine, make an app that does X, etc. This is not exactly the case in theoretical math and never was really.
> For who though?
For math, the value of proving a theorem is often not proving the theorem (which most people already believe correctly to be true or false), but the path taken there, the new math invented, and how it can be applied to other problems. The end result is therefore almost inconsequential in moving math forwards.
1 reply →
Expertise changes perception, not just knowledge.
I have dyscalculia so the math world is dead to me, I can't even add 2 numbers together in my head, but I've read here on HN many times over the years people describe some version of new maths frameworks tangibly changing how they view the world. For me, I went to film school - I have a natural ability to think in movies and pictures, memory is very visual for me, before film school I had a good sense of shape, colour, light. After film school and to this day some 20 plus years later, I mostly look at the world starting with the shadows. Russian speakers, whose language has separate basic words for light blue and dark blue, are slightly faster at telling those shades apart. Speakers of languages that use compass directions instead of "left/right" appear to develop a strong, constant sense of orientation.
These things can only happen through the process, so for who? For people who want to live richer lives, want to think differently, and for you, to be around people who have such.
You mean artificial intelligence? Where did super intelligence come from?
*cite a research paper or something
[dead]
> The output of an LLM is useless without a human mind to comprehend it.
I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.
I find this line of argument unconvincing because it ignores the alignment problem. We only administered mRNA vaccines after a large and diverse group of experts with different backgrounds and experiences came together and said that we should go ahead. We only administered them after several rounds of careful testing to ensure that they wouldn’t have damaging effects. The decision to administer mRNA vaccines for to large numbers of people for the first time was arguable risky and certainly controversial, and yet that’s what happened.
I have no issue with LLMs aiding human scientists, but imo we absolutely would need to understand your hypothetical advancements completely and thoroughly before surrendering any agency to them - the LLM’s risk analysis will not reliably reach the same conclusions as a democratic human society
An LLM could also in theory fake alignment and create technology that seems really useful but secretly harms humanity in the long run. If humans don't understand what it does, that possibility becomes more likely.
mRNA vaccines were created by humans who understood them. You exist within a society (a networked distributed information system) where you accept the judgement of the humans who understand and create this things, which is how you end up benefiting without personally understanding.
If we imagine Super Intelligence, where NO human is capable of understanding, then how would it ever be possible for any human to identify what is actually beneficial or not?
This resolves in a paradox, common to all magical thinking. You can certainly wish that some all powerful benevolent entity will solve all of your problems for you, but it is not likely to work out well.
None of this is new. It is the same delusions as alchemy and the same thing that tales about genies warn of.
6 replies →
> You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.
Normatively, this ought to be true. Descriptively, this is of course false. Our entire society is organized around producing commodities, typically by consuming people as inputs.
Depends if you need to work or not. Plenty of people don't need to work, they can live off their capital returns indefinitely.
For the upper crust of our society, work is entirely optional. Presumably people therefore continue to work for the process or the reward of the doing.
It’s not ‘of course false’, and, to be honest, I can only assume you have little idea what mathematicians actually do.
Most pure mathematicians are no more interested in ‘producing commodities’ than any other academic is. That’s not what the subject is about — at all. The confusion arises because mathematics turns out to be extremely useful (no surprise; it’s quite useful to have a detailed understanding of the basic principles of reality).
Again, this seems to be an alarmingly common fallacy here on HN. As a commenter above observed, pure mathematics (and that is what we’re talking about here) is in important ways closer to the humanities than it is to other sciences.
1 reply →
Not exactly. The input to produce commodities is labour-power (the potential to perform labour). Education or study is the process of producing the commodity of labour-power, the potential labour of a mathematician who can do some sort of specific work.
People don't always understand all the different theorems involved in a proof and can still gain from it. Can even do who layers with things like oracle machines where you just posit it finds a proof and opens up a whole rich hierarchy of stuff. Math understanding will probably get more like physics.
> You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.
That's nonsense. Of course you study them to meet demand.
> you study them to meet demand
nobody who has ever achieved notoriety in any intellectual field ever did it to "meet demand." if your goal is to be an interchangeable widget that produces value as part of a corporate machine that exists for the enrichment of your shareholders, you are certainly free to choose that path, but don't imagine that is the limit of human existence. also, don't be surprise when you are replaced by AI, because it is a superior widget.
1 reply →
People who study STEM for the enrichment of their intellect are privileged. Most of us had to study STEM (or whatever else) because we had bills to pay.
2 replies →
That's not why I studied. Maybe abstractly to prepare myself for the workforce, but the primary goal was to learn.
Unfortunately many countries like the US are very under-developed, and have trained it's citizenry to approach education as job training. This is obviously an abject failure, and probably a contribution to our anemically slow and pathetic growth rate.
sad
>The output of an LLM is useless without a human mind to comprehend it
This is like saying software is useless if the user doesn't read the source code of it. This what happens >99.99999% a person uses software. People want to be entertained or have their problems solved.
Your analogy doesn't work. A "product" in research pure math is not the same as writing code.
2 replies →
> There is a failure to understand that the process is the result
I'm glad to hear people say this now, because for the longest time they've also been bullshitting the public to think that pure math studies would someday lead to some kind of useful outcome.
"transforming the mind" is an exercise in ego-driven self-gratification, no different than getting ten degrees and never getting a job, nor any different than straight up porn or gambling. You have been put on earth to contribute to the progress of humanity, so get dirty and start making commodities. no more useless math-monks.
[dead]
All I hear is cope. I’m very certain that my job can be automated and that I don’t posses intelligence above the models these days. Sure, I’m needed in ways that AI can’t so certain things but I don’t want to pretend like it’s not going to improve on those areas. And unfortunately the Math community need to set aside their ego and come to terms with it. I don’t understand the “but humans are this and that” arguments. AI should bring humility among us. May be every other species got humbled by humans and now we have AI (May be not exactly the AI now but soon)
I know many programmers for whom "developing domain understanding" is an abstract concept (if indeed, it is a concept for them at all). But in the age of big AI, I see more need for this, not less.
At least this is my observation: when my colleagues have been wholesale chucking stuff over to Claude, they've then been confronted with classic XY-Problem shit, poor user experiences, and over-complex solutions (which will mount future problems regardless of whether a person or an agent iterates on that code). Much of this can be solved by actually sitting down and thinking about it, and I mean at a code design level, not just a speccing level.
Many people don't realize that there are more useful outputs to solving a problem than just a mere solution. Obviously if one has a contractor mindset (you don't care about the after effects of a system) then this is of no relevance to you. Some companies promote that mindset, certainly ones that have no broader aspirations then getting acquired soon. That's fine - but many companies actually are about sustainability, and understanding in these places is paramount.
Agree 100%, especially with your second paragraph. The majority of domain experts I've interfaced with in my own domain, along with things I am not an expert in, are typically experts not because of the solutions they build, but because of the things they _don't_ build. It is the cliche "a clever person solves a problem, a wise person avoids it" paraphrase.
But I think what is lost is the value in (a) person(s) who intuitively spots and avoids these problems. My hypothetical example that I have seen play out: The domain expert spots one of these problems and succinctly explains that the problem doesn't need to be solved. Maybe it's a byproduct of bad design elsewhere, or there is a much better solution that avoids it altogether. Once the rest of the team/the lead catches on, either hindsight bias or outcome bias, or a combo, takes over and they don't recognize that all the value all along was in the person being able to spot this situation. They throw that in claude and claude does it's sycophant thing (after expressing that the previous situation was flawless for however long), and they are off to the races right back to automation bias with zero pause for thought...
To me, "domain knowledge" seems to resemble a primitive form of "slop filter" to use the modern parlance. In any field, the majority of possible ideas are slop. This has always been the case. People somehow learn to navigate through the slop, to find things that are worth working on.
Even before the AI era, we were flooded with a veritable DDOS attack of slop, in our in-boxes, meetings, etc. Managers believed that innovation was held back by quantity of ideas, or by the domain experts being "resistant to change." This led to day-long group brainstorming sessions that yielded nothing. Part of the transition from student to junior to senior is learning to filter your own ideas, and to guide others in the search for ideas that are likely to be fruitful.
The criteria for slop filtering have always been subjective, arbitrary, or driven by institutional culture. That's what we've got. Any means of generating ideas produces slop in the absence of filtering.
What will slop filtering look like when the AI can generate tera-proofs per second? What does it look like already for our e-mail inboxes? We are asymptotically trending towards slop filters that look more and more like "ignore everything." I already read none of the business memoranda that I receive. I don't answer my phone. The number of workers needed to read and assess business memoranda is zero.
The slop filter still functions in math. Problems are pre-filtered for being of interest. Somebody expressed enough interest in the Navier-Stokes problem to give it a name and attach a prize to it. There are still plenty of "prize problems." Here are some:
* Quantum gravity
* Improvement of superconductors that don't need helium
* Practical generation of power from fusion
1 reply →
I had an agent (SOTA models, xhigh reasoning, etc.) write and optimize some CUDA code recently. They've gotten a lot better at this stuff. And yet, they couldn't (self-)realize the obvious issue with their code: it was written as if for a typical multicore CPU. What the heck was a (serial) queue doing there? "Work efficiency" as a tradeoff for stalling 20k threads. Its later proposed optimizations were all about "can we get the queue faster" rather than "maybe we should actually parallelize the work for our very-parallel processor".
Funny stuff, as if it were hell-bent on writing a paper rather than actual software.
Sometimes a session just goes bad and is best abandoned or rolled back to before it went off the rails. I've also been using agents to write and optimize CUDA (and the C++ app code) recently and didn't run into anything so braindead. But, I had one session recently where the LLM started making duplicated tool calls and just couldn't stop doing it. Had to generate a handoff and start again in another session and haven't run into that specific problem again. I don't remember specifics at this point but once and I while I have to rollback a session a half dozen turns when its clear the agent had gone completely off the rails.
Like most of us I go back and forth between sheer optimism and fear for the future that AI may usher in. Recently I started vibe coding a fun video game with my ten year old. The experience is different than my work because it’s been such a joy to basically have a personal genie in a bottle help me make some personal art with a loved one regardless of either of our skill sets. The concept the author of this post is arguing now resonates with me more than it would have a few weeks ago. The sheer surface area that AI can create in our intellectual life is limitless and needs humans to explore. There can never be enough of us in that sense. Whether it’s as validators or creators.
Certainly humans can enjoy exploring such things, but will they really be needed?
What does needed even mean? To produce shareholder value? Maybe not. To explore thoughts and ideas humans have for fun? Certainly
24 replies →
I would kind of disagree for personal projects like that. If it’s such a personal project with your young-in, shouldn’t the journey be the treasure? Also, with vibe coding out, custom vibe software/games, and already a lot of games, I would hope you’re not looking for a pay day.
> The sheer surface area that AI can create in our intellectual life is limitless
I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.
The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.
The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.
We're going to have people understand maths more now than ever before.
See proofs never before possible made by someones intuition.
But maybe they wont be mathematicians.
I feel like “giving up understanding” is inevitable.
There’s some hubris in thinking we can understand everything. For truly difficult problems, it’s entirely possible that humans are simply incapable of comprehending why a solution is true. But ultimately the practical value of applying that solution to the real world is going to eclipse our need to understand it.
Math is just the beginning. I see it happening in other fields too, like physics and biology. Many of us software devs have already given up on understanding parts of our own systems for the exact same reason.
Seems like a losing battle.
> For truly difficult problems, it’s entirely possible that humans are simply incapable of comprehending why a solution is true. But ultimately the practical value of applying that solution to the real world is going to eclipse our need to understand it.
This is hardly new or novel, you’ve just described huge chunks of existing engineering disciplines. There is still no complete model that explains why airplane wings work, there are lots of very useful models, but all of them have fundamental flaws where their behaviour completely breaks down in certain very possible scenarios. Notably scenarios where airplane wings don’t spontaneously either stop working or explode.
We also built the entire commercial airliner system decades before we even had computers capable of doing the aerodynamic analysis of airplanes, but none of that prevented us from creating huge engineering disciplines around the empirical data we did have, and slowly chip away at the underlying theories as maths and computing improved over time.
So we’ve always lived in a world where we extract value from systems we fundamentally don’t understand. But that’s never stopped us from working to understand them anyway, and deriving even further value from that improved understanding.
You and the other commenters are all absolutely right, but that’s not really what I was pointing at.
So far our solutions were more or less understood thru some models of reality that we’ve constructed (on our own), which may or may not reflect reality perfectly, and even if most of us never bothered thinking about these models, some people did and they understood them on a very deep level.
But we may be getting to a point where the problems we need to tackle become too difficult for humans to model, or even to notice their existence, like asking an ant how a Boeing 747 works.
Maybe this was always the case but now it seems like we might have a shot at making these solutions useful even if we have no idea what they’re even solving.
You deal with this every time you wake up, or visit a specialist of any kind. You likely didn't make your own clothes, you didn't grow your own food, dam up any rivers for your water supply, build any power plants to power up your phone/computer, nor did you spend any of the deep amount of focus required to build up the knowledge and technology required to provide us these 'things' in life. The only difference is, you didn't pay much attention to these facts because they became mundane. Take any of them away for any decent amount of time, and suddenly you become acutely aware of what it is you have.
The author mentioned what the issue is: a lot of people are experiencing what it's like to 'catch up' to the more intelligent ideas. Everyone can 'get' the intelligent ideas given enough time. But how much time do you have? And he's concerned someone will rush and create some kind of world-ending solution because they're too trusting of a technology they don't understand.
His only solution is some kind of throttle on the advancement of technology, and arguably knowledge (whose?).
That's learned helplessness as a lifestyle.
You observe others behaving poorly and immediately succumb to defeatism, assuming there was no better way.
When presuming, all that matters was the short term economic profit, you simply use the wrong gauge.
Here, long term human survival is at stake, regressing to zoo animals isn't a sensible option.
I think that is the central issue. Is long-term survival actually at stake?
Do you think every Mathematician has a deep understanding of building architecture, engineering, physics, economy, etc., everything necessary to gauge whether something like a 1 terrawat nuclear plant is completely safe in all aspects?
Of course not, so then you have to select appropriate ones. And then you need a process to ingress and egress reviews. Oh what about change? Does that come back to the Math council too?
Oh, we also need global cooperation to pull this off. And then do you trust the people, the selection process for the council (all avenues for corruption are there).
I don't have a solution of my own, so it's not fair for me to criticize the author in this way. I share his concerns.
And it's depressing to think of all the negative scenarios, and be so concerned all the time.
These issues are important and worth thinking about, regardless. Who should teachers teach?
[dead]
Indeed. How many scientists understand how a compiler works? Or know about branch prediction on the CPU? If they do, do they lose sleep because it's non-deterministic? I don't. Seems fine.
In the future I feel one parallel to this is my research area, SAT solving.
People used to solve logic problems by hand, verify logic. Now you pile it into a computer. The problem has been around for a while, no-one fully ‘understands’ the proof of the four colour theorem as a big chunk is computer proved.
We are now just changing (admittedly greatly) what we can put in a box marked ‘checked by computer.
> I feel like “giving up understanding” is inevitable
People didn't stop understanding how to do addition or subtraction when the calculator came out. That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly (by someone, or someTHING that really understands it) and, for complex ideas, time taken to understand them.
> given up on understanding parts of our own systems for the exact same reason.
Maybe this will be a sign of when ASI is achieved. AI researchers keep saying they don't fully understand how LLMs work - maybe when ASI arrives, it can explain that fully.
> That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly
You should try being a teaching assistant or a tutor. "Everyone can be taught everything, it just depends on the teacher" can only be said by someone who never tried this in practice. Just as not everyone can learn advanced math, the best mathematicians also have their own limits and there are things they won't be able to understand or comfortably navigate. Not just because it's a lot of material, but because it's complex inherently.
"We’re gonna need a lot more mathematicians."
What the article really says is that we're going to need much smarter mathematicians. That is not possible for puny meat-brain humans. Humans are close to their ceiling. AIs are just getting started.
In practice, we're probably going to hit that limit first in IC design. I once went to a talk by the Intel engineering manager who headed the Pentium Pro effort. That was the first superscalar x86 CPU, and it took about 5,000 engineers at peak to design it. Getting that many people coordinated on one thing was a real achievement. Then Intel stayed with minor tweaks on that design for years.
We're soon going to be seeing designs of even greater complexity cranked out by AIs. No human will understand them at the gate level. Reading AI-written programming language code is bad enough. Reading AI-written Verilog may be beyond human comprehension, except in small sections.
> Humans are close to their ceiling. AIs are just getting started.
The whole point of mathematics is to vastly exceed that natural ceiling by gradually building a framework for understanding. In fact it’s wrong to speak of a ceiling altogether. If there were a ceiling, we’d have hit it long ago.
AIs have already swallowed the entire history of human thought, but apparently they’re ’just getting started’. I can only assume you don’t know what mathematicians actually do.
Abstractions can certainly enable this kind of "telescoping" effect of understanding, but not everything can be compressed with clever abstractions. Some things are inherently incompressible and there is no neat and insightful short explanation for why it is true, just a massive proof, but it may still have provably good properties for building a chip or power plant.
We have naturally only explored the mathematical universe in the parts where telescoping via clever abstractions can get us. But there is much more. Being able to juggle more things in your mind at the same time can have qualitatively massive benefits.
Information theory and proof theory, algorithmic information theory etc has of course explored this.
5000 engineers for the Pentium Pro?? Bob Colwell, chief architect of the Pentium Pro, says 450+ people with over 400 design and validation engineers. Source: "The Pentium Chronicles", pages xvi and 2.
> Humans are close to their ceiling.
Hopping on this train: the human ceiling 100 years ago is now advanced undergrad material in mathematics. I see no particular reason for this to change, especially with everyone in the math research pipeline pushing to compress the difference just as always before.
> Humans are close to their ceiling.
If Math Academy teach 10 year old kids calculus, I doubt that.
Rule based Mechanics of taking derivatives and finding intervals is quite different from understanding delta epsilon proofs. I have no doubt 10 year old can do the former quite easily, the proofs on the other hand? Depends what you mean by “teaching calculus”!
> I once went to a talk by the Intel engineering manager who headed the Pentium Pro effort. That was the first superscalar x86 CPU
The original Pentium was already superscalar, with its asymmetrical U and V pipes.
The Pentium Pro added out of order execution via register renaming. A true achievement, indeed.
And as for the more general point you are making: computer chips have been far too complex for any single human to comprehend for decades. I left NVidia after working there as an archutect for five years, barely understanding anything about those behemoths.
[dead]
> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Why would mathematicians be the best at this or even able to contribute? Engineers and physicist seem like a much better choice. Mathematicians tend to not bother with messy things like whats physically possible, or human consequences etc.
Agreed, it's a very strange example. What percentage of tenured academic mathematics researchers or PhD students work on power plant related things? I know there are applied mathematicians who model things like structural integrity of buildings and prove theorems about this and there is deep math in radio communication and electric grid design etc. (graph theory, coding theory etc), but this is just a small part of math.
I find this discussion odd. 50% of people mistake this guest post for a post by Tao himself. The post is by Amit Sahai, professor for computer science and math at UCLA.
Tao has been posting a flood of guest posts that stress inevitability and coping. Tao is fully invested in AI but needs to have the appearance of a broad discussion.
Sahai is of course exuberant about AI but wants to keep UCLA enrollment numbers high. The comments on Tao's blog are far less friendly than here, because by people see through the game that is being played.
Just look at the latest industry friendly post from Tao, where he pretends that it somehow summarizes the discussion so far:
https://terrytao.wordpress.com/2026/09/25/iciam-statement-on...
For my whole life I have never been subjected to an industry coordinated advertisement campaign that ruthlessly harnesses YouTubers, TikTokers, professors, open source people who all go in lockstep.
Tao, Gowers et.al. will go into history as the professors who ruined math.
[flagged]
You got the responses before you edited out the embarrassing stuff, liar.
[flagged]
3 replies →
The main reason to learn something is actually being able to communicate in the language of that subject. There are complex ideas in math that cannot be easily captured by the language of other fields. Pepole who don't study math cannot even understand what a worthwhile goal in math even is or how it could be useful to other fields.
You can't simply prompt a model to be "better" when "better" isnt even properly defined
> To take on this responsibility, we may need to broaden our view of what a mathematician can contribute. I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs.
> Our ability to understand difficult and unfamiliar ideas may become one of the most important contributions we can offer to society. We should be willing to bring that skill to problems far beyond our usual research interests. [3] Doing so asks us to expand our sense of our vocation.
Am I reading this wrong, or is he talking about what (mathematitian) Data Scientists have been doing for years? So he is basically saying that former Data Scientist that have turned into prompt/software engineers should go back to being data scientists.
In any case, people should stop trying to fit AI in the previous status quo. What we need is curious people, that is what we have always needed.
A few centuries ago there were no "mathematitians", there were mathematitians/philosophers/artists/physicists all in one person. So its not like "mathematitias" is something that has existed for millenia.
We need curious and ethical people.
Maybe AI brings back the age of a well rounded scientist/philosopher. I know this sounds counter intuitive because the article is saying that we cannot keep up with the AI.
I believe he’s advocating for mathematicians to understand the novel concepts introduced with each breakthrough rather than individuals understanding data stemming from empirical observations.
A lot of the conversations around AI in the past years have been “we will delegate almost all work to AI, but the most decisions will be left to humans.”
I’m fairly confident that the opposite will be true. The most important decisions will be made by AI, and humans will only be left to guide decisions as a matter of taste. AI has the ability to be impartial, and immutable. You can endlessly probe and reason its decisions.
This isn’t true of our current human decisions. Where we see red tape, bureaucracy, rent seeking, status quo. AI will see through these human constructs.
I do like the idea that we will need more mathematicians, physicists, and scientists to understand the discoveries of AI. That might be the best possible outcome of AI, but I fear the opposite and we are painting ourselves into a corner of which we no longer have the knowledge to sustain ourselves and society itself collapses.
> red tape, bureaucracy, rent seeking, status quo. AI will see through these human constructs.
Why would an LLM care about eliminating any of these things? When you say "red tape" and "bureaucracy" you are, in that very statement, applying emotional labels to what are actually just systems working as designed. There are the rules, so we follow them. LLM decision-makers will happily enforce and perpetuate bureaucracy and red-tape. If the rent-seeking is legal behavior then they will fully protect a big corporation's right to engage in it, they don't care about the feelings of the little guy. If a similar situation was decided a certain way in the past, then yeah let's decide it the same way now and maintain the status quo; LLMs very often think that way, they don't care about human ideals like "change" or "progress".
> AI has the ability to be impartial, and immutable.
Which AI has this ability? LLMs definitly don't.
With an open set of weights, fixed seeds and parameters, recorded inputs and feedback loops, the entire AI decision making process is auditable.
It also lends itself nicely for the ability to adjust the AI systems when deficiencies are found. You can tweak the AI systems to align itself with the desired output, and check for regressions, much like tests in a codebase.
Human decision-making lacks this thoroughness. I’ve scarcely come across rigorous documentation of decisions and how they were reached professionally.
I recommend you the book the rational optimist.
Not saying you’re wrong you just can’t be so certain
the dynamics of discovery on a population level are also not fully understood. there's no doubt in my mind that the machines will grab all the low hanging fruit in terms of novel ideas through mechanical recombination of existing public science, and they will do so faster than humans can. what is not understood is whether they will plateau.
today they benefit from human willingness to share publicly and improve ai systems, but once those systems start to threaten their livelihoods, will those dynamics change? can the machines push the frontiers or is the global network of human creativity and tenacity necessary?
setting aside the dynamics question, it is known that LLMs have limits wrt the theorems they can prove without human input, see my comment here: https://news.ycombinator.com/item?id=49756412
> This is a guest post by Amit Sahai
Not Terrence Tao post. Beware.
it seems like most of the recent posts on Tao's blog are not actually Tao's?
If we accept that AI is going to do all of these things humans will be superfluous and will just be optimized away. It looks a lot like we see the birth of silicon based life by the efforts of carbon based life. Carbon based life will die and it will not even be because AI decided to kill it, it will be because feeding and watering it was less important than other concerns. Bacteria may continue to exist, though.
Is that already economic delamination? As in the singularity extracts the most valuable ressources (new thought generators) from the traditional economy keeping mankind alive and stable- towards itself?
As of now the ai needs hands, and maybe more importantly, cognitive intentionality which thus far only arises from living beings drive for homeostasis to survive and reproduce. If ai can have this same intentionality is a big unknown.
> But we are now entering a time for humility: a time when all of us are going to know what it feels like to be unable to keep up.
Humility is the wrong word. We didn't feel it when the steam engine was introduced, so why now?
>humility
Fear? Ai is an abstract (for most people). A steam engine, you can touch. And, it doesn't replicate.
The problem is not that we fear the AI will do maths we can do. At least that's not the problem. Whether the AI can write better code, devise better proofs, etc, I still want to do it. The thing that worries us (the generation trying to enter the job market now) is just our livelihood. Food, rent & accomodation while I do the work. "Who's gonna pay me, and for what ?". That's the question I want answered. And please, if the response is some demeaning work like reviewing the gigantic ball with 1% meaningful insight, 90% slop and 9% other people pillaged work that LLMs will be dumping at my door all day long, then society may need more mathematicians, software engineers, etc but no one is gonna risk going along that path.
> "Who's gonna pay me, and for what ?". That's the question I want answered.
Unfortunately that’s a question you’ll have to answer for yourself. Also, people are going to risk going along whatever path gives them work as many today already do.
Russell once remarked that all of mathematics would be trivial to a sufficiently intelligent being. It builds conceptual tools for limited minds, that lets them understand far beyond their natural reach.
In a world with ASI, having that capacity is vital.
Ha, are you invoking something he said prior to discovering Russell's paradox? Which came about because he wanted to show that all of mathematics can be derived from logic about sets, but instead he showed that it couldn't be.
Russell's paradox was 1901. Likely afterwards.
As someone with a math background, I don't see Russell's paradox as some interesting mathematics beyond set theory. It's just an example showing that one should be careful about defining sets.
1 reply →
> "If AI fulfills its promise, we will encounter more beautiful and consequential ideas than we have ever seen."
It will be as it always has been: these ideas will further enrich the rich, at the expense of everyone else. We'll have our first quadrillionaire, while the masses are debating whether the minimum wage of $7.25/hr should be bumped up.
What a doomer way of thinking. Scientific advancements has positively improved the lifes of people in the past I don't see how scientific advancements accelerated by AI would be any different.
It’s interesting that you won’t say it’s wrong.
> Another approach is to follow that word, heresy. In every period of history, there seem to have been labels that got applied to statements to shoot them down before anyone had a chance to ask if they were true or not. "Blasphemy", "sacrilege", and "heresy" were such labels for a good part of western history, as in more recent times "indecent", "improper", and "unamerican" have been. By now these labels have lost their sting. They always do. By now they're mostly used ironically. But in their time, they had real force.
>> The word "defeatist", for example, has no particular political connotations now. But in Germany in 1917 it was a weapon, used by Ludendorff in a purge of those who favored a negotiated peace. At the start of World War II it was used extensively by Churchill and his supporters to silence their opponents. In 1940, any argument against Churchill's aggressive policy was "defeatist". Was it right or wrong? Ideally, no one got far enough to ask that.
What You Can’t Say https://paulgraham.com/say.html
1 reply →
One question: LLM might answer correctly many logical and math problems, but I don't see any guarantee that when the LLM context receives new information the answer could get worst. Security in LLM answer is not a monotone increasing function of context size.
If we assume that trusting a model to execute an action is an ongoing exercise, since its trustworthiness is often discovered by the user organically as models develop, should trustworthiness be measured at the level of the model, or at the level of the human intent behind triggering it, whether explicitly or implicitly?
> The ideas and opinions presented here are entirely my own, but GPT 6 Astra was instrumental in helping me draft this note.
It really looks like Tao is on the path of accepting and embracing AI now.
Maybe he is, but you are quoting Amit Sahai, the author of this blog post.
Oops :D
Are you willing to pay for a lot more mathematicians?
Or it as with other professionals: shortage means shortage of cheap labor.
> Imagine that a future AI system proposes a radically new design for a one terawatt nuclear fusion power plant
This is an unfortunate example to choose, being as it is entirely confounded by the canonical illustration of the https://en.wikipedia.org/wiki/Law_of_triviality.
How is that relevant at all?
How is it not relevant? The construction directly contradicts what Sahai imagines happening, and the worked example is the nuclear power plant.
Life is all about tradeoffs. Mathematics have no tradeoffs. We need more engineers.
Mathematics can give for example theoretical lower and upper bounds for complexity of algorithms. And that information is very valuable for tradeoffs. For example we know that many algorithms have optimal average complexity or asymptotic behavior so they can be a good tradeoff in certain scenarios. Also knowing that some problems don't have a perfect solution allows you to accept essential tradeoffs
Life is all about purpose. Mathematicians are instead concerned with meaning. We need more philosophers.
I keep clicking through on these because I am curious about what Terrence Tao has to say, and then they are always guest posts.
I think understanding the output of AI will become more and more likely impossible. It's not necessarily a bad thing, if we can benefit from AI produced knowledge, even if we didn't get there ourselves.
The point of the post is that what you call "AI knowledge" is not knowledge.
Always confused by text website that tries to prevent people from select or copy or right click.
"Imagine that a future AI system proposes a radically new design for a one terawatt nuclear fusion power plant."
I propose we enact a new law that makes it so ai engineers can only fly on ai designed planes powered by ai designed engines flown by ai. Maybe then we'll get some real progress out of this slop producing crapware or the problem will be solved a different way.
It would be nice to have more mathematicians, but we don't need more. Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
Even if that happens, what’s the point if there’s no human involvement? AI doing math for math’s sake? And doing what with it?
I think it is like saying what is the point of playing chess when you can never beat stockfish?
What is Magnus Carlsen going to do when he can't beat the computer?
It seems like a category error between humans using tools and humans building tools.
There is not much point in trying to figure out a better chess engine. There has never been a better time though to want to learn chess.
I find the idea that the computer will discover mathematics and humans call it a day rather ridiculous. As if humans will not then spend their time understanding and incorporating the ideas from the computer.
Alphafold is a better example. Alphafold is only bad if you spent your life trying to solve protein folding. But even if you did, that is the same person who is the most setup to reap the benefits of the unlock in the pragmatic application of protein folding.
We don't figure out how to get machines to harvest corn and then spend all day sitting around eating corn in between naps.
> AI doing math for math’s sake? And doing what with it?
Stuff! Inscrutable stuff, maybe, but that's not "doing math for math's sake."
> AI doing math for math's sake?
Yes, why not? And, of course, AI doing math for AI.
We may not be needed forever...
I don't understand how to make a modern CPU. I'm not involved in the manufacturing of it. From my perspective, there may as well not be any human involvement. I can still use the resulting chip (in an larger system of other things I can't make and wasn't involved in) to argue with you on the internet.
It becomes another abstraction, really. As long as we can use it for something useful, it's still valuable.
3 replies →
I think a better analogy would be comparing to an ancient human instead of an ant. An ancient human would have none of the basic abstractions that we take for granted today like literacy and arithmetic, so it would be very difficult for them to even attempt trying to understand quantum mechanics. But I don't think it's impossible because our ability to learn by stacking abstractions is basically endless——so far as our health permits at least.
We've been those ants for a million years, and only in the last 100 did we start to wrap our heads around quantum physics. We are the purpose behind creating LLMs. There's plenty in the universe we don't understand, and it's very human to keep striving to do so.
If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.
Don't write us out quite yet. :)
Or maybe ants and humans are qualitatively different. Maybe there's a critical mass of intelligence where you can pretty much understand anything, and maybe humans are past that threshold. I don't know that for sure, but I don't think we're anywhere close to hitting fundamental limits to our ability to understand the universe.
Need? We don't need lots of things, including computers. We did well without them for hundreds of thousands of years.
We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.
That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.
> Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
This makes no sense whatsoever.
We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.
If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.
>and that means we need expert humans to define and verify them.
And this will be true for how long? 3-4 months?
4 replies →
you arent a real mathematician unless you lay around all day thinking about shapes
Mathletes, assemble!
> I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs.
I can’t afford to be one of your credentialed reserves, Amit. ADHD screwed me over in early life or I’d have breezed through that dual masters in cognets/compsci by 2003, and now that I’ve stabilized my brain and am burning through years of school at a rapid pace, my country’s socioeconomically ruined — I’ll be lucky to escape with my accounting degree as my school is visibly being sucked into the vortex every year I progress. The only hope left for me to be what you need of us is to self-study, but without a degree I’ll just be treated as a crank or an AI proxy/puppet if I slip up and talk about interesting math with anyone, so what’s even the point of taking that path? I originally pivoted my math skills into systems theory and process diagnostics instead, which of course now everyone has kicked to the curb and replaced with AI. I’d have made an excellent Susan Calvin — I was working to be a cognitive tech with diagnostics, zeroth law, and group psych specialties, AMA! — but the financial investment to provide the runway to take that lonely, dreary six year slow as molasses slog through maths that universities think is somehow an appropriate teaching velocity (six months for precalculus alone?!) in order to earn the mere chance to have my resume rejected by an AI firm that uses AI hiring and and can’t tolerate someone with a strong moral position regarding societal harms is a very bad choice, whether you use simple probability or game or systems theory to evaluate it. Taking that quarter-mil-plus burden as loans in the hopes of employment at the other end in a field actively having its social, reputational, and moral fabric being ripped apart by AI? That’s not just a bad bet, that’s chasing fool’s gold at the end of a fading rainbow in a desert mirage. So, with sincere apologies, while you’d benefit from having me on your ‘reserves’ list (I have written testimonials spanning some twenty-plus years to that effect) I’ll never come to your attention as a support tech or as an accountant, and I accepted that outcome years before anyone else realized this need for cogsci mathematicians with a teacher’s specialty of analogy-building and the ability to disregard interpersonal nonsense to focus on the needs of societies. Better luck with the next generation, though!
I think this is the most beautiful letter on the subject I've read all year, it brought a tear to my eye. It's like reading those famous STEM letters/essays from history.
I agree. I believe it could be the best piece about AI ever written. The themes of lost dreams reawakened, sobering up of egos, sci fi technological development, the role of humanity. It’s a weird feeling to see that it’s actually really happening.
Can we add guest post to the title? A lot of people here are attributing this to him incorrectly.
> Imagine that a future AI system proposes a radically new design for a one terawatt nuclear fusion power plant
I imagine:
* Massive budget overruns
* Year and year of delays
* It ending up wasting more energy than it produces, or just not working, period
* Politicians disclaiming responsibility and blaming it on "the AI"
* The contractor companies profiting immensely over the entire period, and ending up not having to worry about maintenance, warantees, etc.
TT is politely and gently trying to deal with the ego of many mathematicians, namely to teach them humility, which they will need to design their specialized maths AI models(agents?). Well, dunno if it will work out anything interesting, but he is giving a try.
I wonder if the current maths models, if any, are able to use formal solvers in their 'reasoning'. I wonder if a natural language interface is really that efficient, maybe a pure formal language hinted with intuitive "tokens". I remember the time I was learning real maths: "elegant", "brutal", "strong", etc were somewhat meaningfull.
It’s not Tao. None of these guest posts are Tao.
> [This is a guest post by Amit Sahai. This blog post was initially written in a different file format and converted using AI. — T.]
No matter how good your case is - if you use AI slop to spam text, I will not read it. At the least they admitted to this sneaky slopness, which saves me time. I'd wish everyone would do so.
[flagged]
[dead]
[dead]
Mathematicians and physicists are just an organized cabal who are shamelessly profiting off of artificial complexity they create to maintain job security. The Yes people.
Watch me get downvoted some more. Nobody cares about the truth, we now know thanks to our beautiful president Trump telling us about that and the Fake News.
There's, like, one, two at most, mathematicians who are really thinking outside the box. The rest are like sheep. Or cult members with mass psychosis.
Waste, Fraud, and Abuse people are coming to academia, which is so hopelessly parasitised by the left.
"I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs. " -- I can't be the only one who thought about the scene where the scientists ask Deep Thought the ultimate question...