Comment by pizlonator
3 months ago
What I can’t get over is that there have been exactly zero software breakthroughs since vibe coding started, other than vibe coding itself.
Claude is amazing, that’s true.
But if it was as amazing as this article implies, I’d expect some breakthrough outside of AI itself.
Rewriting a Zig program in unsafe Rust? Not a breakthrough. Finding a bunch of security vulns? Maybe that’s sort of a breakthrough though it’s underwhelming and possibly just a net negative. But like if I rolled back to using software from 2023 then life would be ok.
Maybe we just need to give it time, and sometime real soon, we will all be amazed by such a breakthrough? Who knows
Maybe my bar for what constitutes a breakthrough is lower than other people's, but all of these seem like breakthroughs to me:
NLP as a field saw huge shifts. NLP tasks that used to be complex and inaccurate can now be setup very easily and quickly using structured outputs from LLMs, often with greater accuracy.
A small charity I help with has now been able to build their own website to manage their day-to-day operations. It saves them a lot of time, and it was vibe-coded using Manus. I don't think people appreciate how much room there is left for bespoke software to have big impacts on small organisations that can't afford to hire developers. The cost for software like the one they made has gone from 10s of thousands of dollars to $10/month and volunteer hours.
My brother has recently been setting up Cowork to do an automatic review of contracts before human review, and he said it is far more diligent than people when it comes to routine things to check. This is another huge breakthrough for not just efficiency, but the quality of work.
I really don't think we can discount AI finding bugs and vulnerabilities. If you care about code quality and keep up review standard, LLMs can help you write more robust software. AI has found a huge number of bugs for me before they hit production, including potential out-of-bounds memory accesses and segfaults.
ChatGPT has 1 billion MAU. People are now getting life advice, financial advice, and mental health help from chatbots at a scale and cost that no human support network could match.
> ChatGPT has 1 billion MAU. People are now getting life advice, financial advice, and mental health help from chatbots
Personally not the kind of breakthrough I'm psyched about
Yeah, the thing that worries me is that an LLM can be guided to agree with any premise and will rarely ever take a hard stance.
2 replies →
Well, you're not twisting yourself into knots to identify breakthroughs. Try harder!
> ChatGPT has 1 billion MAU. People are now getting life advice, financial advice, and mental health help from chatbots at a scale and cost that no human support network could match.
That's terrifying.
You realize that's terrifying, right?
Definitely, it is quite an extreme change. But the upsides of better access to support and advice are huge, even if the potential downsides are scary as well. This feels like one area where we need better transparency and regulation due to how much ChatGPT and others can affect people who listen to them.
Its in a weird space right now.
These models are actually extremely good but they are far from an intelligence unto themselves. Truth is if someone told you they could build these things 5 years ago, you d write them a check for a trillion dollars. Problem is once we got them, we realized they are not all that. Its like a mecha suit in a universe, where mecha suits are abundant and cheap. Someone has to climb into them everyday and put in the work for it to be effective.
So now the skeptics are saying this technology is overrated. And the optimists are accusing the skeptics of moving goal posts.
I think we are learning in real-time what intelligence re. humans is as we go along.
Humans only what they know, until they acquire more information about what's possible.
The goal post narrative is stupid to begin with.
Humans have goal seeking behavior. LLMs don’t. You could maybe call the combination of LLMs and the RL-based harnesses somewhat “intelligent” in aggregate, but the problem is that it’s not “general” intelligence like these labs want to argue, since it’s by definition only good for the set of problems the RL part has been trained to solve, which is a subset of programming problems.
> Problem is once we got them, we realized they are not all that.
Isn't this just the hype cycle? [1]
Fake edit: I know its not a perfect model.
1: https://www.gartner.com/en/research/methodologies/gartner-hy...
The problem is what they can do is rapidly expanding. Software development is becoming increasingly hands off.
If they get to the point where they're smart enough to make tasteful code decisions based on stakeholder input... we're cooked as a profession.
Most of the skeptics exist because of the grandiose claims made by the AI companies saying pure hype marketing bs. If this was just a tool, discussed at the scope of what the tools can actually produce and do, there would be sensible discourse about it.
I am doing a solo project that is pretty big, meaning it is not something I could vibe code. I can do alot with AI that I could never do on my own, but I am not seeing several mulitples improvement in my productivity. I spend so much time doing what I call "AI wrangling", trying to get it to do what I want. Claude is writing all the javscript and python code, but ultimately I am programming in English. What is good is that it is effectively a very high level computer language, where the agent can implement a lot of underlying code with a short English description, often. But many other times it takes a lot of work to get what you want.
I measured an ~8x increase in the number of commits I've been pushing, and I've actually been trying to restrain myself. I could do a lot more if I stopped reviewing and editing the code. I think it's got more to do with my executive ability than raw productivity though. AI essentially cured my ADHD by making the execution of my ideas virtually painless.
LOL "I measured an 8x increase in the number of commits Ive been pushing" is an absolutely useless statement
2 replies →
I'm building a memory safe programming language with a declarative concurrency model that's close to release.
There is ZERO chance I would ever be able to complete it on my own.
I doubt it'll get traction, but if it doesn't, I am pretty confident a future language will take the ideas for polymorphic synchronization and profile-guided optimization.
It has an easy version/mode of compilation that makes Rust's affine ownership accessible like a high-level scripting language, and it can progressively become more strict, where the compiler does ~99% of the work for you, and you just pick options as it finds issues (that it explains to you like you're 5) along the way.
Along the way, I also built a suite of tools that helps identify complexity better than anything I've seen (which was necessary to get the LLMs to be able to unslop themselves and write something that actually works).
I doubt the Ruby community shrugs it off, but time will tell.
How do you know it’s actually memory safe?
1 reply →
I have the same experience, though I feel myself getting better at wrangling over the past few months
I spent years in the early 2000s trying to get a computer to read unstructured PDFs and TIFF images (mainly invoices, either scanned or electronic). Limited success, we always had to get a human to look at them in the end.
We implemented that in about three days earlier this year, just by feeding the files to LLMs. And it's good enough to not need a human to check.
I get that this isn't a "Computer Science breakthrough" in the sense you mean, but it used to involve a lot of hard CS to try and solve, and now it doesn't.
Maybe I'm looking through rose colored glasses, but software that writes itself seems like a pretty big breakthrough to me.
That goes straight to my point: then why hasn’t the miracle of automated coding led to breakthroughs outside of automated coding?
If the only breakthrough is automated coding with no outside consequence then it’s just masturbation
Probably because AI coding has only worked at all for a couple years and has only gotten good in like the last year?
The rate of improvement has been fast. Maybe it’ll plateau soon, or maybe we’ll have LLMs improving themselves rapidly. At this point it’s too early to say.
I don’t remember where I heard it, but there’s a saying that people overestimate how much can be accomplished in a year and underestimate how much can be accomplished in 10 years.
If we get to 2030 and still people are wondering where the breakthrough is, then I think I’d be agreeing with your skepticism. But I just think it’s too early to judge that yet.
3 replies →
N=1, but Claude etc. have made a huge difference to my life personally.
Built a bunch of software tools to streamline my small ecommerce business - while also running it - and things have turned around from "losing money and ready to pull the plug" to "looking at our best financial year on record" in the span of about 8 months.
I could imagine it wouldn't make a huge difference to the life of someone deeply entrenched in a traditional tech role, trying to get an extra 9 of reliability in a service or roll out a new carefully planned and QA'd feature.
But for tech-adjacent people, it gives us something "good enough", instantly, and basically for free.
That doesn't include the other things I've got it to do (gave Claude SSH access and got it to successfully debug a hang on my Ubuntu server, chucked Codex in a folder full of financial data and got it to find every piece of misclassified payroll transaction data)
Genuinely the biggest breakthrough for "casual" tech users since Excel.
1 reply →
What would qualify as a breakthrough for you?
What is your bar even? automated coding has changed the game already.
Strictly speaking, it's modifying itself. Although it would be an interesting challenge - can an llm create a new llm from scratch?
No, it probably can't during our lifetime at least—but it can sure modify itself to avoid antivirus detection, which is _just swell_.
1 reply →
Which is funny because people have been using LISP for that since 1960.
Which is what makes putting an LLM inside a lisp so much fun
It's pretty crazy that a company like Anthropic no longer needs to hire Software Engineers, because their software engineers itself. If that's not a break through I don't know what is!
edit: it looks like I was wrong and they're still hiring many software engineers. Not completely sure why that is just yet.
The arguments against AI assisted coding used to be "only for toy projects", then at some point it became "no dignity", "joyless". Now it's "no new breakthrough" apparently. All in the span of maybe a year. I say it's made tremendous progress.
Then where is the big new non toy project created since vibe coding became a thing, that couldn’t have been created without ai?
don’t know if this qualifies as big in your book, but there are some well marketed advances here:
https://deepmind.google/blog/alphaevolve-impact/
Openclaw
I make one (small) almost every day. Admittedly the reason that couldn’t be done is because it would take time that I don’t have but 1000% every day something is written by AI that I use that would not exist if AI didn’t exist.
I don’t publish them - but they’re put into use in production and they provide a tangible benefit that would not exist otherwise.
2 replies →
What does a breakthrough look like?
Some examples:
- The first web browser
- the first web browser with images
- typescript
- react
- rust
- Fil-C
- doom
- quake
- the anamorphic VM, and its follow-ups like HotSpot, and even competitors/copycats like J9, V8, JSC, etc
- Fortnite battle royale
- Roblox
- thefacebook
- ChatGPT
- Claude code
I know that’s quite a range and that’s intentional.
Anyway, I think we’ll know it when we see it.
Reading through that list. None of those were breakthroughs when they first came out. It took time, in some cases a long time for them to become good.
- Completing the full CL implementation of Emacs or better still finish Lem.
- Complete GuileMacs, the Guile implementation of Emacs. As AI is supposedly much more capable than Humans, it would be great if the above mentioned implementations are even more efficient and feature rich than Emacs!
- Something like Android (maybe even a clone?) with the Java Layer removed and replaced with CL and with Linux kernel still intact. Basically CL over Linux as opposed to the Java over Linux in Android.
- For fun, an implementation of the Lisp machines' OS with Lisp all the way down though Assembly is allowed for critical pieces. It should be a full blown modern Desktop with equivalents of what users expect from a modern OS ...
The LLM+Harness mostly helps with execution.
These are new products (generally) and that's a different class of problem.
It is possible that since LLM+harness helps with execution then we should see more experiments.
2 replies →
Massive productivity gains.
Yeah.
To play devils advocate, computers didn’t translate to massive productivity gains until long after businesses adopted them. There was that quote from ’87: "you can see the computer age everywhere but in the productivity statistics"
Maybe we’re seeing something like that right now with AI?
Who knows man
6 replies →
How about a Windows file browser that opens in less than 5 seconds.
That sounds like a your-system issue. I hit Win+E (admittedly on an old Win10 box) and it instantly pops up an explorer window.
2 replies →
FilePilot has been a thing for a while now
1 reply →
> exactly zero software breakthroughs since vibe coding started, other than vibe coding itself
Generative AI is meant to be a mimic - Richard Sutton
https://x.com/RichardSSutton/status/2061216087744946656
The breakthroughs in mass state surveillance are coming, never fear.
What does a software breakthrough look like in your opinion?
If you get yourself to define it, maybe you'll find it achievable :)
Solved a bunch of Erdos problems.
What would qualify as a breakthrough for you?
openAI has how many employees and the chatGPT app has 1 billion MAU
Vibe coding is the breakthrough. There's always been "no-code" solutions to problems in various business domains, but they were invariably janky, underpowered, and/or overpriced. Now we have a way for domain experts to go directly from ACTUAL natural language directly to implementation in a real programming language, fully automated, in minutes or hours. How is that not a science-fiction level breakthrough? In 2011 if anyone had said that would be possible "in 15 years", I think most professionals at the time would not have replied with "yeah it's coming but your timeline is off". It would have been "you have no fucking idea what you're talking about".