Comment by qoez
3 hours ago
Without a thriving mathematical community to point out these things it would have stayed broken. With automated math that community as tao pointed out is at risk.
3 hours ago
Without a thriving mathematical community to point out these things it would have stayed broken. With automated math that community as tao pointed out is at risk.
This is not a problem unique to the mathematical community.
What about software developer community? AI has eliminated the need for junior software engineers. Almost no one is hiring junior software engineers. But companies still need senior software engineers. Without junior engineers how will there be senior software engineers in the future?
What is the solution? I don't think the solution is to say AI progress in software, mathematics etc. should be halted.
TECH! PRIESTS!
https://warhammer40k.fandom.com/wiki/Tech-Priest
Oh, yes... the progress... You measure it by LOC, right?
I hope we discover a solution before innovation totally stalls, but I do not think any solution has been found yet. And I agree that halting AI is unlikely to be the solution because even if we halt the big companies China for example can still do AI.
Some people hope AI will get good enough in a few years that it can innovate without human experts. Maybe? But that remains to be seen.
It does remain to be seen.
By the progress of AI from ChatGPT to now is horrifyingly fast.
Trendlines and basic reasoning point to a most probable future where the AI is superior. We can’t just say “that remains to be seen” because the alternative is the least probable future.
Anticipate the change and act prior.
I'm not old enough to have gone through the revolution that was programming languages that got increasingly more abstract and decoupled from the metal, but surely there's a lesson that can be learned from that era?
Those abstractions are deterministic. LLMs are not.
AI models and tooling has advanced significantly in the last 6 months. Give it a couple of years and the companies will not need senior software engineers. They will need someone to steer the AI, maybe. Why would anyone need senior software engineers?
> Give it a couple of years and the companies will not need senior software engineers. They will need someone to steer the AI, maybe. Why would anyone need senior software engineers?
Right, companies won't need software engineers. They'll just need someone who can use tools to produce source code and maintain the generated artifacts, plus make domain-specific technical decisions like "what should the system do when two users update the same record as the same time" or "how should the system behave when a message in the queue cannot be processed".
We really oughta come up with a job title for these people.
1 reply →
Because the models do what you ask them. "Claude, do this thing, be thorough, no mistakes" is not a good prompt.
[flagged]
I think this will be a self-correcting problem once the damage is done, and the only cure is more cowbell, I mean - TIME.
In the age of extraction capitalism where building sustainable, profitable companies is not the goal, no one will care.
Hopefully, there are some sustainable, profitable companies that keep a low profile now and will take over after the "self correction" happens, and whoever invested in them will become very rich.
Tons of corporations rely on the fact that human interaction within software production will be as minimal as possible, and no one actually cares who trains the seniors of the future.
> Without a thriving mathematical community to point out these things it would have stayed broken.
And that's a bad thing. If the math community didn't exist or was weak, OpenAI would still benefit from the prestige of these results they were forced to withdraw. Withdrawing these papers has harmed OpenAI's investors, and that's totally unacceptable.
> With automated math that community as tao pointed out is at risk.
Good to hear. The problem they represent needs to be addressed.
OpenAI: "Here are the answers to every math problem. Now mathematicians are obsolete. Just send the prize money and prestige to Sam Altman. BTW gonna need some mathematicians to check these answers."
There's plenty to be concerned about without making up problems like OpenAI claiming any prize money.
Have you got a link to someone pointing it out? It looks like they're still going through formal proofs so likely found the problems that way.
Yes: https://x.com/ElliotGlazer/status/2108026240582246600
So someone ran a different LLM to find an issue they'd find anyway during formalisation? That's not the same as relying on thriving community.
15 replies →
That’s not proof though is it? If the original LLM output is fallible surely the LLM review of that output is also very much fallible?
2 replies →
The fact that humans are needed to correct mistakes is only an ephemeral status quo that is being eroded away as we speak. We all can see it happening in front of our very eyes. AI is getting better.
It is pointless to call out the little wins humans still have because again, those wins are temporary.
We need real concerted effort into asking: what is the point? For me the only answer I came up with is: fun.
> With automated math that community as tao pointed out is at risk.
If they can be automated, they are not necessary. If they are necessary, they won't be fully automated. It's a pretty simple experiment to run, the math "community" should bear with us. Darwin would be proud.
This statement assumes local maxima don’t exist and greedy short term optimization always leads to long term benefit.
> assumes local maxima don’t exist and greedy short term optimization always leads to long term benefit.
Even if your stated assumption was baked into the original comment, which is doubtful: the historical record shows that we will keep relearning The Bitter Lesson and each community will pretend what they do for a living is exceptional and immune because of xyz. The screams will get louder when the "greedy" and "dumb" automation comes knocking and it turns out nothing was truly immune or "nuanced ".
Getting some new hobbies may be in order, it's a Brave New World.
1 reply →
“If they can be automated, they are not necessary.” That’s a pretty interesting take as eventually everything could be automated.
> “If they can be automated, they are not necessary.” That’s a pretty interesting take as eventually everything could be automated.
It's great that we're starting to see the light at the end of the tunnel, and will some day achieve a perfect market without humans. If you think about it, all the market really needs is a people to own everything, everything else can be automated, and all those annoying human workers can be eliminated.
I guess what OP meant is automated and with 100% accuracy.
You've abandoned every part of yourself to the siren's song of efficiency and automation, huh?
To witness an arson and rejoice reveals an ugly kind of sadism.
Will it also be "arson" if OpenAI dump on us solutions to more practical problems, for example, a blueprint for a better photolithography machine, or a nuclear power plant?
1 reply →
[dead]
I'm sorry but you don't actually need the mathematical community to check any of this. You just need autonomous verification, which already exists at scale and speed vastly beyond the entire human mathematical establishment. OpenAI simply rushed these out without completing that for every paper. These mistakes have nothing at all to do with the mathematical community and are 100% just the result of market pressure incentivizing speed at 1,000,000x the pace of human mathematicians. Frankly, a couple mistakes, trivially found not by humans but by humans using AI, is almost completely irrelevant. Human mathematicians have essentially nothing to contribute to this effort aside from prompting verification agents, which any of us can do if we cared to spend the time (most of us don't).
> You just need autonomous verification, which already exists at scale and speed vastly beyond the entire human mathematical establishment.
An absolutely ridiculous statement. There is a vast amount of mathematical knowledge that hasn’t even been written down, much less formalized.
There is no such thing as knowledge that has never even been written down a single time by anyone or anything. Those are simply called ideas, they are not unique to humans, and they are more likely to be wrong than right compared to anything OpenAI just produced. It would be bold and almost certainly spectacularly wrong to assume that multi-trillion-parameter AI models could not possess their own such ideas, generated during the training process as part of their internal model of the world. Beyond that, if we assume that such knowledge does exist in humans, its marginal value is clearly nearly zero now, as AI systems without access to it are vastly outperforming all human mathematicians combined by multiple orders of magnitude.
6 replies →
It's amazing that people long long ago figured out so much math.
Is the risk here like "people will fund math research less because of AI"? I agree that would be bad, and we should try and stop it (along with e.g. funding for the humanities, which is in a much worse place than math!), but I'm not sure OpenAI are the right people to be mad at.
But they are the ones releasing an unverifiable (no model release) and massive and unchecked body of mathematics into the public while making exaggerated claims about its capabilities to replace human work. And they are the ones doing it in advance of a fractional sale of the company to the public.
Based on what I've heard from my math friends in academia, every talented undergrad who was set on going to grad school for math has switched to something like consulting internships or fintech, even if they were really passionate about math, because they don't want to spend another 7 years or so just to wind up jobless.
This has always been the risk with majoring in math. AI is making it worse but there was never a time where studying math didn't have an extremely high opportunity cost.
1 reply →