Comment by zarzavat
12 hours ago
I worry this is looking at where the ball is now instead of where it's going. The recent disproof of an Erdos conjecture should put to rest the idea that LLMs will reach a skill ceiling before they reach superintelligence.
I believe we are headed for a world of superintelligent AI where LLMs are much better at logical thinking than humans, the same way that chess engines are much better at chess than humans.
In that world there's really nothing humans can offer in terms of logical thinking other than their humanity itself. An 8 year old with Stockfish can beat Magnus Carlsen, and an 8 year old with Codex (and daddy's credit card) will be able to beat me at software engineering.
I don't buy that at all.
It doesn't matter how great the LLMs get, the act of creating software using them will still require a great deal of skill.
Most people just don't think in terms of software.
Try asking a non-developer in your life what their dream software would be for their work, or their hobby. If they don't have what Nilay Patel calls "software brain" I'd be surprised if they came up with something actionable.
(For more on software brain see "THE PEOPLE DO NOT YEARN FOR AUTOMATION", which makes the point I"m making here but much, much better: https://www.theverge.com/podcast/917029/software-brain-ai-ba...)
You could give a non-developer the smartest LLM in the world and they wouldn't be able to create GitHub with it, because creating GitHub requires an enormous amount of understanding of what software developers need from a cloud source control tool.
Sure, you can argue that the LLM "knows" what GitHub needs already and can guide their human-user to that, but why would a human-user who doesn't understand the domain ask an LLM to do that in the first place?
> Try asking a non-developer in your life what their dream software would be for their work, or their hobby. If they don't have what Nilay Patel calls "software brain" I'd be surprised if they came up with something actionable.
I've posted this in numerous comments because I think it bears repeating: there are tech-savvy non-developers who are actually building and shipping stuff with AI. I personally know a few who have been successful in acquiring initial customers.
You can say "but their apps won't scale", "their apps aren't secure", etc. and you might be right but these criticisms ignore the fact that most human-built software suffers from issues around scalability, security, etc. What AI in the hands of a relatively tech-savvy person is capable of is building functional, usable applications that are pretty decent compared to what you might get if you paid an experienced contractor tens of thousands of dollars to build.
A whole generation of young people has grown up with the internet, smartphones, etc. They might not be trained software engineers or have a "software brain" but in many cases they probably have a better intuitive sense for digital product design than a 30 or 40-something engineer who has been staring at an IDE for the past decade(s).
> there are tech-savvy non-developers who are actually building and shipping stuff with AI. I personally know a few who have been successful in acquiring initial customers.
It'll shake your world, but tech-savvy non-developers were building and shipping long before AI.
> they probably have a better intuitive sense for digital product design than a 30 or 40-something engineer who has been staring at an IDE for the past decade(s).
Because developers only stare at IDE 24/7, and never interact with anyone besides mother who brings tendies to their basement? What am I even reading?
1 reply →
> there are tech-savvy non-developers who are actually building and shipping stuff with AI
I absolutely believe that. I think those are people with "software brain" who are on their way to becoming real developers.
By the point they can write apps that are secure and scale... they'll have learned enough about software development to be employable as software developers. They'll be part of a new breed of developer who never memorized the syntax of a programming language, but they'll still be at the starting point of learning a HUGE volume of other stuff that's necessary to build good software.
If we want to stay employed, we need to be notably better at building software than they are.
15 replies →
> I've posted this in numerous comments because I think it bears repeating: there are tech-savvy non-developers who are actually building and shipping stuff with AI. I personally know a few who have been successful in acquiring initial customers.
I mean, sure. But there have always been people teaching themselves to program too. In the end it's a pretty small population.
I personally don't see that much sense worrying about this scenario because if it comes true then it doesn't really matter what I do.
If building software (and even programming, as the basis for it) was just an expression of logical thinking, we would have cornered it long time ago IMO.
But then again, logic is really a lot more discrete and well defined and easily expressed with traditional computing than LLMs are (which are probabilistic systems instead and as such require large knowledge bases).
We can observe that at a couple hundred billion parameters they behave similarly to a point (in the sense that they can produce similar results), but the challenge is really in understanding the problem's multifaceted structure and competing needs and priorities.
Are you confident in putting a timeline on this prediction?
One of the reasons I'm increasingly skeptical of this prediction is that I've now lived past a few of the dates I heard people put on the achievement of this level of superintelligence in previous years.
Chess and proofs only work as comparisons to the extent that you can find parts of your job that share their key property: A solution is sought to a problem that can be stated with relatively little information.
What prompt would someone have used to get a superhuman coding agent to output the Linux kernel or GTA5?
Before you accuse me of moving the goalposts, that's not my point: The examples are there to help think about what humans would still need to do to build complex projects even if the coding itself was perfectly reliable.
Both the Linux kernel and GTA5 contain a large amount of incompressible information; humans thought long and hard about how to design them, i.e. about what that thing they were building was even supposed to be.
You don't understand, Claude 69 will be able to one-shot GTA6. You NEED to buy into the fearmongering and anxiety.
> In that world there's really nothing humans can offer in terms of logical thinking other than their humanity itself.
By your logic anyone who's not in the top 10% of intelligence can't offer anything. The world keeps spinning.
> An 8 year old with Stockfish can beat Magnus Carlsen, and an 8 year old with Codex (and daddy's credit card) will be able to beat me at software engineering.
That's just nonsense, nobody will work with 8 year old (it's illegal, to start with). Go touch grass.