The gap in shared understanding of LLM capability is widening

> But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point

It seems we’re perpetually in this state.

I think the point being made is that when you're feeding outputs into inputs, both 93% and 90% rapidly collapse to 0% without some way to recover.

First off, the 90% to 93% success rates are not step-by-step failure rates between successive steps of pursuing a solution. They are overall end-to-end success rates on extremely difficult problem sets that most humans would score close to zero on, and most humans with sufficient training and experience would score significantly less than 90%. And they are problem sets that grow successively harder as AIs get successively more capable. A few years ago AIs were being tested on high-school level math problems. Now they are being tested on problem sets that challenge people with PhDs. Or problem sets that humans are (so far) incapable of solving.

Compare that to humans. How often do you think humans get it right the first time? Almost never, for sufficiently complex problems. And how often do humans get the overall result wrong, without realizing it? Also, almost always for sufficiently complex problems, or sufficiently ordinary people. And what percent of the general public would be able to get even one question right in a problem set based on Math Olympiad problems? What percentage of mathematicians are capable of solving even one Millenium Problem?

Furthermore, you have to take into account that AIs do not particularly use output to input architectures. I frequently see self-corrections to plans when I'm developing using AI assistants, and plan changes are sometimes major. ("Oh I see. I need to revise step 4 of the plan." -- Claude paraphrased). Clearly current-generation AIs use more than input to output feedback; they also use supervisor/client strategies of AI and collaborative networks as well. And often provide self-correcting infrastructure, varying from "Claude: Oh. That generated a compiler error. Let's fix that.", to "Claude: One of the test cases is failing. Let's see what's going on.". And that self correcting infrastructure allows AIs to try again and again until they succeed. (Not unlike what humans do when solving difficult problems).

As a member of the 0.2% (but not the 0.00006%), I have to imagine that frontier models, when applied to research topics, are using most of those 10,000 instances to investigate alternate hypotheses in parallel. Which is also something that humans do across entire disciplines. Based on my personal experience with difficult technical and commercial and scientific problems, I sincerely believe that most breakthroughs are not made because whoever made them is more brilliant than the other hundred or thousand or dozen qualified people who are working on the same problem, but are made because they happened to be lucky enough to start with a somewhat random hypothesis that luckily turned out to be the right one. In case you were concerned that this is a sour grapes point of view, yes, our team was the one that came in first.

So even 90% may be a very big deal, for problems that human get right about 0.1% of the time. Or even for problems that AIs get right about 0.1% (but have enough instances available to pursue 10,000 hypotheses in parallel).

p.s. Ironically, it really looks like humans score significantly less than 90% on the "is 90% good" problem.

> vibe coded Photoshop clones. Yeah, they work [...]

I think what people (sometimes) miss is that building something is one part of the work, but keeping it running, developing and refining it is a whole different story. And I think all those vibe-coded software will not stand the test of time.

Photoshop is 36 years old now and it wouldn't be the most-used, most-established solution for professionals if Adobe didn't do great work over all that time.

Not really rooting for Adobe here, dislike their subscription models just as anybody else, but I think you'll have to recognize that they kept it running and never stopped.

The public has yet to produce a result of comparable impact to HuggingFace, to my knowledge. It's not really about the model, it's more about orchestration infrastructure. Seems like we've hit the point where you don't necessarily need a smarter bot, you just need an arbitrary number of them working together.

No doubt the unreleased models are better at the things that LLMs are good at (coding, hacking, math) than what's been released, but there is still zero sign of their intelligence becoming broader and more general, and knowing how these models work and are trained there is no reason to expect them to become much more general unless changes are made.

The marketing message being pushed by Anthropic (& also OpenAI) is that these narrow math/hacking/coding skills are representative of the model's intelligence, when that seems to be the exact opposite of the truth - these narrow human/super-human competencies are entirely atypical of the models more general capabilities.

I think we'll see an unbelievable upheaval as companies figure out that while the technical aspects of some jobs can (and should) be automated, the interpretation of those technical aspects and other higher order, but not strictly inherent processes can't. By upheaval I mean companies closing because they replace staff with AI, and lose things along the way because of it. They won't see it until it's too late, because of the pile of money they're saving.

My example comes from maintenance in machine shops. My dad's siblings were almost all employed in machining in some fashion. One uncle was maintenance for a local large company. They decided they could move maintenance from in-house specialists and contract that work to a multi-state massive company; anyone can follow a maintenance manual right? All you need are the proper certifications and everything is cool, right? The cost was less, and the number of maintenance staff on site actually went up. Great for the owners.

But what they lost was consistency and understanding in the process that eventually led to closing the company. My uncle was there every day all year long for decades and could hear and feel in his feet when something was going wrong with a machine regardless of the maintenance schedule approved by the company that manufactured the machines. The new outside company didn't pay as well, and didn't have great benefits, which meant that they cycled through staff very regularly. No one was there consistently for more than a year or two. Even though they were technically following the manufacturers recommended maintenance schedule, machines went down. Machines then stayed down for days or weeks longer than they should've, because the expertise to repair either wasn't there, or the person leading the job would quit and they would have to cycle in a new maintenance person. They lost jobs, customers took work other places. 8 years later the company closed.

There are lessons in this kind of thing, but nobody learns them.

I believe that the imprtatnt bit that many will come to understand ,maybe after costly mistakes is that 'tasks can be automated, jobs cannot(yet)' one still needs the discernment to know what the sequence of tasks lead up to meaningfully.

It kinda reminds me of the story of the Toyota process , while their car product lines were/are heavily automated the process still relies on the smalll teams on the line to spot/solve and iterarte over mistakes/inefficiences then add the refinements back onto the automated tasks.

Similarly althose stories of radilogists going jobless have not panned out , as interpreting slieds is a 'task' that a radiologist does.

Semantics , yeah but it does shape the way of thinkings regarding jobs/economy and not to mention the whole anthropomorphizing by the AI labs and boosters. In essence the wordings can/could introduce a healthy dose of reality/skepticism to the wildest predictions regarding job losses etc.

This is already an issue I face both with clients and personally where I can say something authoritatively, but they’ll always go and just ask an LLM.

Sometimes it’s in my favour. LLMs estimate very high prices and long timelines for software projects.

If it’s a legal / regulatory issue it’s usually wrong.

Your concern about higher-ups valuing being told they are correct vs actually being correct is exactly what I saw at my last job (one of the largest companies in the states) and now at another smaller firm. It was really disheartening because I do use qwen a lot for personal stuff and claude at work, and have been able to multiply my productivity by only having to understand a ticket and then code review instead of doing all the intermediate steps, but a lot of people have let it go full self driving and just tell it "implement ticket ABC-123 and raise a PR" without checking the output at all.

I wholeheartedly agree. To clarify my position, I don't think having these amazing models mean we should replace everyone, quite the opposite.

I do think they are close enough to the stage where people will be replaced, even though it's not the right or correct thing to do, and that we need to move on from the "denial" phase quickly enough so our societies can adapt / regulate / legislate while not too late.

This, 100 times over. And it's already happening.

Which isn't necessarily a bad thing. Smart leaders / entrepreneurs will realize this and then use human input where required. The rest will be cleared out after the computer has made enough mistakes for them.

>I know how to interpret their needs and find the solutions that fit for them.

Without additional explanation this is indistinguishable from the role of a technical product manager.

[dead]

Sizing the steps matter. Disclaimer up front: we are all figuring it out as we go along, there are no hard and fast rules that fully generalize. You just need to try and develop your own feel for it.

In my case, for some types of work - which sound similar to things you mention - I started slowing AI down intentionally. Software development, if it's something simple/common and thus "in-distribution", LLMs can zero-shot complex projects just fine. But in more R&D work, I found that errors crop up quickly and accumulate, and end result of "go and just make X" is usually bad.

Now, the problem I found with usual workflows for this kind of work is, agents race ahead and very quickly, I can't keep up anymore. So I reworked my workflow to make AI work in small steps optimized strictly for maximizing my ability to validate the outcome, and minimize time I spend doing it. So now, the thing (say diagnosing a performance regression of some complex codebase run on a cluster) may take me a whole day, where "unchained", agentic swarm would do it in 30-60 minutes - but, at the end of the day, my method is near 100% guaranteed to be correct, with every intermediate step vouched and reusable for future work, whereas "agentic swarm" approach, that hour of work, 90% of the times I'd just skim and throw away and reprompt, because I couldn't even begin to guess which part is correct, which isn't, so in the end, it was just a fast way to waste lots of tokens.

I see many people have complex systems involving skills, agent swarms with roles, agent factories, 20 termux panes and whatnot. Maybe that's the way, I don't know. Nobody knows. But it's not necessary to start getting value from LLMs. I therefore recommend my "lazy person's basic approach":

- Throw a task at LLM

- Check result, see what you like or not like about it

- If you just can't keep up, ask it to do smaller steps; if you don't understand, ask it to explain

- Just do that a lot, and don't be afraid to make both absurdly large and absurdly small asks

- You'll quickly get a feel for where AI can help you, and where it's not yet worth the time handholding it

Opus 5.5 is superhuman at most white collar work. At this point, not using it (including for health/law) is basically malpractice.

Yeah, and I keep wondering why.

They can hire more people and still be massively augmented by AI.

I'm reminded of an interaction I had with a non-technical user. They knew ChatGPT can translate text, and so, to remove spelling mistakes from their (Dutch) text, they would first ask ChatGPT to translate it to English and then ask it to translate it back to Dutch. Mind-blown when I said "you can just say 'fix the spelling and gammar', no need to translate"

This is the level that many non-technical users are operating on.

Using LLMs well with human language requires precision.

There are only a few professions that are intrinsically precise with language.

No surprise we're seeing adoption in software programming and mathematics.

I'm currently seeing a lot of developers and AI industry insiders that do not distinguished between the capabilities of their LLM Agent and the added capabilities of the agent harness. They think the capabilities of the agent harness are within the LLM, and in that confusion they are trying to get their agent to make them an agent harness with all these capabilities. But LLM agent harnesses are brand spanking new, how to make these harnesses are not in the training data of their agent, these developers are more or less vibe begging their agents to make them a harness, so it hallucinates the harness architecture. I am seeing a "agent harnesses" that are delusions, the these developers are not strong and experienced enough to realize they are literally doing the worst work they possibly could... And these idiots are the AI leads.

> think it is a soft skill that only hours of practice can make you good at Yikes

>"not cost effective to automate" (which is effectively everything)

There still seems to be a wide gulf in automation capabilities and things that require human dexterity, such as doing things for the infirm. Even just cooking and laundry seem quite a bit away. I guess the original tweet does mention this:

>Capability peaks in domains that are digital, verifiable and economically valuable.

Driving is the most impressive one so far, in my opinion.

It is very good at programming. Because programming is a lot of language, and these tools were also made by programmers and post trained to be good at it.

But I assure you it's even "better" (not really but the task is so much easier) at making your spreadsheet look nice and compare actuals vs forecast for your budget than a complicated programming task. It will just one shot it, and it will be objectively correct, all in a very short time. But these non tech people don't know they can use an agent, they have asked one question in the chat interface with 0 context about what they're trying to do, and conclude they are not good enough.

I'm not saying everything is replaceable by AI - and by the way I don't want anything to be replaced by it, but I believe it will happen - just that some people don't realize yet how "trivial" their job is to modern models.

It is particularly suited to programming. But most business processes are basically a program that is executed. Especially in modern times, where office clerks have less and less decision-making options.

The prices are dropping fast. At some point one of the labs will decide they have a model that is good enough and they will burn its weights into silicon which will make the model much cheaper still.

> demolished software and mathematics because those have well-developed verification that's easier than doing the thing itself

They are proving useful for the parts of those things that have that verification, but in neither case is that the whole of the field. In maths, for example, we have seen LLMs find answers to some interesting problems/conjectures, but haven't yet seen them pose any similarly interesting problems/conjectures themselves because the criteria for a "good conjecture" are unclear whereas the criteria for a resolution of an existing conjecture are usually supplied by the conjecture itself.

lowering the cost and speeding up delivery will mature technology in other fields leading go the disruptive verification that ai needs

> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers

Isn't this a massive alarm bell? That the capability hasn't done much for consumers yet that isn't basically toys? Although I suppose the pitch was always B2B "you can automate away your expensive white collar workers" and it's not clear how widespread that is actually happening.

What is the incentive to try? Any such application, if profitable, would immediately be replicated. If one person can use agents to do something profitable, anyone can use agents to easily copy it. Hence you have Anthropic locking down their models and removing features customers can access, to try and protect their own slim advantage.

With this stuff I immediately wish I was more economically literate. Bare with me. Let’s say one of these labs can use agents to replicate work otherwise done by humans. This proves that anyone with agents can eventually do this, unfortunately I don’t think its a moat like a platform (like facebook) provides, given that everyone is leapfrogging each-other constantly. Thus, it kind of becomes a commodity / low margin business?

Depending on the type of output, the value immediately (?) goes to the electricity cost of the LLMs and the cost of materials. So: more “efficient”, but without a big margin, right? So its cheaper for the workers that remain working, other workers are displaced, and capital owners now have a super slim margin business which can be instantly replicated at a place where material cost are lower (?).

Thus, it almost seems like “value add” by humans would in this simplified view be vastly diminished, and all that remains is resource wealth. Pop the materials in on one end and let the AI manufacture the product. Great news for the third world assuming the USA doesn't invade them.

Also, whose the consumer? Or are we going back to a strange medieval time where consumers are not of concern?

I don’t know what to think

Maybe raw intelligence is not the bottleneck for what can be accomplished with the current data available in a form agents can access it?

> The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?

Most peoples jobs do not involve hacking other services in large scale fuzzing/whatnot attacks.

> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.

It does not seem to me that tech industry or its investors care about consumers even a little bit. Consumer preferences dont matter once you have enough or raw economic power, I guess.

This weekend I had some time with Opus 5.5 and GLM 5.3. I went over some findings in a library of Claude Mythos 5.1. These were minor, low-severity bug fixes and Mythos provided a PR to address them. Its PR was functionally correct. I still spent ~5 hours looping over them until I was really happy, even though a fully working PR already existed.

Some of the differences were:

- Mythos recommended salvaging damaged labels. I pushed back: data whose structure is broken shouldn’t be trusted

- My simpler more readable test caught a regression that Mythos’s more elaborate test missed

- Mythos allowed invalid values, citing real-world data that I could not reproduce, while I decided to follow the spec

- Mythos kept the old error types. I made every kind of malformed input raise the same error, and accepted a small, documented breaking change for anyone who relied on the old ones

Overall, I much preferred the final version of my PR to the initial draft Mythos provided. However, getting there involved quite a bit of time messing around with the code, as well as some back and forth with Opus (which was valuable).

This is a recent example that matches my overall experience working with current SOTA models (the ones mere mortals like me have access to).

TL;DR: I don’t think AI already has better judgment than (most) senior people. Obviously, that might change with future models. But that’s a hypothetical scenario. It might as well not change anytime soon.

do you really want AI to make life and death decisions? do you really want that power concentrated in the hands of the few who own the AI systems?

Last week I was dealing with a bug where we were triggering some behavior in a service in all requests when actually we really shouldn't be triggering this behavior for one of our callers. It just so happens that this caller sends a unique flag to the service.

The suggestion from frontier models was to have our service check for this flag and use that to decide whether to trigger this behavior, even though the flag had nothing to do with this behavior semantically. I'd expect this from a junior engineer. Correct now, but almost surely to cause a bug in the future as the codebase evolves.

These tools are incredible and can achieve things I would never have imagined five years ago. I have yet to see their judgement exceed mine.

>AI already has better judgement than many senior people in some tech domains, on top of better skills

It has better judgement in math and coding, which have infinitely generatable training data; if you ask it a domain-specific question about something like sales or marketing, its answer is often horrendous.

> AI already has better judgement than many senior people in some tech domains

How would you quantify this?

> "AI doesn't have judgement" is a mirage, that will shatter within the year. AI already has better judgement than many senior people in some tech domains

No offense, but if you noticed it has better judgment than you, consider that you might not be a "senior people in a tech domain"

>managing pricing on my Airbnbs

so, scamming your customers?

Sure, they fix these embarrassing things as fast as they get pointed out.

Today I gave a free tier AI a photo of a burning ship posted on Twitter, and asked it to confirm it was an oil tanker. It said no, it was a bulk grain carrier, and gave a specific ID. I pointed out that the bulk grain carrier it had identified was painted black, not red (like most liquid carriers) like the one in the photo (as well as other obvious differences).

That is just one example. The free tier models continue to make very dumb mistakes all the time. Maybe the top secret models, too dangerous for the public, are able to distinguish a black ship from a red one. Who knows.

Who are these "AI cultists" that you have asked your strawman generator to conjure up out of thin air?

Yeah, have the "experts" permitted your usage of this parable to illustrate your point? We must know this.

Hmm, that's poor API design. If the big companies can't standardize their inputs and responses, then the small projects aren't going to either. Sounds like there are lots of small issues like this that will roadblock AI-powered automation for some time.

I wonder if in response to this companies like google will build a sort of agent facing layer to the app in question. So rather than you building an agent to attempt to wrangle pupeteer to click css elements as a user does, it calls some apis google exposes to certified gemini user agents and these contain exit codes and can be run as script like instructions not e.g. the shortcomings of browser agent based methods such as having to wait for all content elements to actually load and the obfuscation arms race some devs have towards exposing these elements towards headless agents.

[flagged]

> Slack puts {"ok": false} inside a 200.

I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.

I'm normally not the cynical type, and I am appreciative of Karpathy's efforts to educate people on developing language models, but this is exactly how it reads.

Every once in a while, everyone needs to touch grass.

As used in LDPC decoders?

My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory.

Question to any AI researchers out there: Can a layer in a trained AI be treated as a form of "matched filter"? Is that a concept that is used in the understanding AI: to view it as a cascade of matched filters? The next question is "what is it matching?"

> we teach machines to learn

I feel like this statement is too anthropomorphizing. We don't teach a computer anything. We write normal computer programs that can process data(training), turning it into what is essentially a configuration file(weights/parameters) for another normal computer program(inference engine). Upon receiving a request, the inference engine is invoked and passed the input, then it processes that input and its configuration(weights), producing an output. "Thinking" is just feeding the output back in as an input, allowing the model to revise it and refine it before it's returned to the user.

There is no teaching, nor learning. There is just data processing. There is nothing happening between requests to the inference engine. It's a relatively simple system altogether, but when the training data set is large enough it produces unbelievably good outputs.

With this in mind I really don't understand how people think these programs can be conscious. They're not doing anything fundamentally different than any other computer program. It's just crunching numbers. You might as well argue that a computer game or 7zip or a chess bot or whatever is conscious.

I know you specifically didn't claim they're conscious, I just thought it was worth mentioning because I think it's ridiculous how many people will seriously argue this as if there is any difference between "ai" and other programs. There isn't. It's just a set of instructions for doing computation, as computers have always done.

3D rendering is actually quite similar. There is a bunch of files (input/configuration) describing 3D meshes, textures, lighting etc which are fed into a rendering engine which processes all that data and turns it into a 2D image. Similarly to AI, once the input files are sophisticated enough, this can produce mindblowingly realistic and beautiful images, and you can run it in a loop to make video or make it interactive, moving the camera around in real time etc. But at the core of it all is just a bunch of files containing a bunch of numbers, and a program that reads those files and does computation with them.

Tbf OP said they have the tools to understand what was going on, not that they could build a model to compete with OpenAI tomorrow. EE (especifically DSP and communications) uses a lot of the same building blocks as ML so yes it does prepare you.

You are right in that the emergent abilities come from humongous amounts of training data and compute which only a handful of companies (and the people working in them) deal with today. I'm sure there's a lot of secret gotchas which only arise at a certain scale so one cannot self-study for them. Companies also guard these wherever possible because of economic incentives. But that applies to most other engineering disciplines. You can take a course in civil engineering and have enough knowledge to reason about why a skyscraper will not fall apart in the wind; doesn't mean you have the ability to build one.

But it’s certainly enough to know that it’s not “conscious”

From memory the entire filter was updated in one symbol period (ie. global)

The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.

Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?

That's very true, and I think you are probably more right than I am (in thinking this ability will _not_ be evenly distributed).

I do wonder what happens when:

"Help me find a tool to do XZY" is responded to (by an LLM) with "Here are a couple of options but I could also build one for you...". But at the end of the day no everyone wants to be a "Builder" or manage it. They are happy to exchange some money for someone else to handle it for them similar to the "everyone can cook but people still go out to eat" concept.

My colleagues write 10x less code for the same tasks (when they don't (ab)use AI) and don't reach past abstraction boundaries so that it's possible to actually reason about and review their work. When the problems get hard, they apply more care and attention to detail. Even if LLMs had the same bug rate per SLOC (IME it's higher), and even if those bugs were the same severity (IME they're worse), you'd still expect 10x bugs just because they can't be arsed to write a reasonable amount of code.

Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.

This is a different argument from Karpathy's prompt-and-forget one.

I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.

I am shocked how many bugs AI finds in the gray bearded wizards code at work. Just tons of stuff I didnt catch and I assumed wasnt there.

> You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?

Not sure that is the position you're arguing with. I would refuse to skip code review for any colleague, no matter how trustworthy they were. This is the behaviour we are expected to accept to get productivity gains, when the colleague is an LLM.

That's not really my point... my point is that it's an unsettled debate between speed and caution, and devs are not unanimous on it, unlike the claim of the original post that seemed to think everyone was reporting the same gains.

The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.

Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.

A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.

Last week, on my pet project that I am using a harness to almost entirely code (so I'm by no means a Luddite), it/Astra decided to dump ALL application state into a single row+column in SQL as JSON. That is a mistake that I would, and a completely new grad, would _never_ make. It is profoundly incompetent.

This week it did it a second time on an unrelated project.

"Some mistakes" is putting it unbelievably lightly. It is shitting out complete and utter garbage, and if you think it's doing a reasonable job (or a job that can be fixed later by a more powerful model), you're either not paying much attention, or your own ability has already heavily atrophied.

Karpathy put it well: "gap of understanding of LLM capability," but he's seemingly in one of the less fortunate groups himself.

Imagine if other engineering disciplines just started accepting that everything they built or worked with would be only 98% correct.

Over time, the errors don't "work themselves out together," they compound. Everything would stall in a mess of unreliability and chaos.

I don't quite get what's the argument you're making here. What's in contention is whether or not the 0.2% is in agreement.

Are WE the 0.2%?

Because if not, what's the point talking about the merits of AI's competence in of itself? Accomplishes nothing, we're he outsiders. And if we are, then either you're fishing for disagreement and the post you replied to is actually correct, or you get unanimous support to your statements, and establish... Nothing, the in-group could be delusional in a way that does not actually function as an argument against the whole "So where's the software? We're the AI led market disruption instead of the AI-led shovel makers?". The punchline was about the clients being unhappy.

This attitude is exactly why AI bros don't make successful software lol. "Where's the software?" This. This is why there is no software.

Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.

It is a for-real science.

This. So much this.

Are AI models incredibly powerful tools? Yes.

Do they have the ability to telepathically sense what the various stakeholders in the development process actually wanted? No.

I feel like this was always true to an extent, now product and design are just more tolerant of it because leadership is cracking a whip over their heads because things are supposed to be much faster “because ai” so things like pixel perfect hand offs, adhering to existing design system, making sure you are aligned with the brand, copy is reviewed with marketing go out the window. Now everything is shipping in hackathon mode with an eng driving decisions they really should not be.

I should frame this for the wall behind me when I run meetings with my team. It’s the core problem with AI in coding.

Even projects that you could have done without having to look anything up?

In my experience, it seems to mostly save time on things you don't really know that well, whilst depriving you of the chance to learn new things.

Opus 4.5 in November of 2025 was the first model that would save more time than it wasted for an experienced developer. Everything that came before was a cute parlor trick. Sometimes it could complete a function without shitting the bed, but mostly it was just an annoyance creating messes you'd need to clean up at least as often as it did the right thing.

If you weren't mostly writing code by hand a year ago, you weren't writing very good code.

Maybe AI development is partially so popular because to some people that situation looks the same regardless of who is writing the code.

Meanwhile I'm looking at code that frankly does a bad job of showing why it even exists and nobody involved can tell me why. But it does something vaguely similar to what a prompt said it would I suppose.

A big problem for whom? The AI providers? If people don’t have a use for AI stuff that’s fine

Why so sure they will get better and not just plateau? They seem to be doubling in cost each time with less notable gains, the economics seem increasingly shaky, and potential problems with model collapse and context window size has not been solved afaik.

"Most people using it for work already pay API rates"

Citation?

I deal with bugs frontier models made literally every day, and people are making specs. You have the escape that "maybe they weren't proper specs" but this is a no true scotsman fallacy.

We have a continuous process of updating skills, specs etc to try and avoid all these bugs they write, give them context about why its the wrong thing to do.

This process is actually a lot of work, continuous, and can likely continue indefinitely.

>proper spec

No one has any clue what this thing means....

Boy, AI chuds sure love inserting the language of inevitability into their cute little polemics.

> the field as we knew it is dead

As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral

I'm thinking scbench.ai, and the fact that future token costs may become more prohibitive to just use it for everything.

For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.

There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.

Also, so many vibe coders don't seem to know that you need to keep third party elements patched.

Lol, “functional programming, etc” is a great line. I would dismissed it as some lazy handwaving usually, but from a longtime account it reads very wry.

Definitely the best comment I’ve read from the demoralization patrol in a good while.

People keep talking about AI slop but then I go maintain our largely human written code base and it's like worse than any slop I've seen from my AI usage. This is with code reviews, tests, and a supposedly strong engineering team.

That said AI feels a bit alien.

My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.

It's so bad. 3 out of 4 answers for me are wrong.

The other day I asked which trains stop at a particular station and it pointed me at a map that isn't real.

It very, very routinely points at articles in its RAG "citations" that simply don't support the (modestly likely to be correct) assertions.

It routinely recommends Mac software to iOS users until you ask it to confirm there's an iOS version.

It still claims widely used software has functionality that is non-existent, based on roadmap/wishlist/feature request/unreleased fork type pages.

It is better than it was but it is not adequate yet.

To give a specific example of where I've seen this utterly fail... A few months ago one of my pets had an unexpected reaction to a new medication. It was late at night, well past our vet's hours. Google's AI response was basicaly "They are going to die, rush to an emergy clinic now".

But nothing else matched that. I was looking up all kinds of resources and that hadn't been the conclusion from any of them. It turned out Google AI was just wrong and put us in a bad spot for even looking at those results.

A large portion of the time I google something that I know a lot about the AI summary is misleading if not outright wrong. Frightening when you consider how many take their so-called AI as gospel.

God damn. I respect you for being honest, but… god damn. That’s a pretty low bar.

There are, for two years already when OpenAI released o1-preview:

From wikipedia: "large language models that are trained to solve tasks that require several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, make use of extra computations, and enhance performance when increasing the number of parameters, training examples, and training computations."

That's the same thing as the parent but stripped of worry.

I see this from everyone I know who works at Anthropic, they seem trapped in some sort of AI psychosis where AI is on the verge of ending the world.

Meanwhile, back in the real-world, most people I know use Chatgpt a few times a week for totally mundane shit and even more people never use AI at all.

And in my world, the latest frontier models still do moronic shit in the codebase all the time.

Meanwhile, my friend X at Anthropic tells me he fears that AI is going to have a terminator moment within the next year or two...

It's bizarre honestly.

Funny way of spelling "delusion".

Lester Bangs: what are you listening to?

William Miller: Stillwater

Lester Bangs: Stillwater?

HANGS UP

Lester Bangs: Fucking kid’s doing drugs

I stand corrected. Then more broadly, there is a level in the "funnel" that comprises of people who have only tried ChatGPT and not Claude.

ChatGPT also has much stricter guardrails and will avoid going anywhere near large swaths of human knowledge.

I'd strongly implore anyone who uses ChatGPT for domains other than programming to try the same prompts in parallel on Opus 5.5 (Extra) for a few days. The difference is often staggering. ChatGPT is mostly just a coding and automation LLM.

I work at a company that holds very sensitive data and I don't think there will ever be a non-US-made open weight model in our walls.

I love luna but unfortunately it needs to be much, much faster in tokens per second to compensate for lack of intelligence.

> Economics do now work out in my favor mostly.

That means, it is mostly cheaper to do it with Copilot etc. and not alone, right?

you might be right, but without evidence of this we can't know.

Much less than seeing new software products being shipped, we havn't even seen attempts at them being shipped, or built for that matter, again excluding this sort of meta software around the AI itself like harnesses and tooling.

We could speculate that attempts have been made but not shipped, and the details of why not being revealed, but then we're just back to it being talk with no demonstrable stuff happening in the world that backs it up.

I've always struggled with doing basic life stuff. Paying bills on time. Remembering birthdays and to buy presents. Doctor appointments. Setting up play dates for the kids. Getting my teeth cleaned regularly. Booking in vaccines for the kids. Renewing passports. Booking flights. Etc. It feels like so much of my life is consumed by this useless administrative bullshit. I LONG for personal agents which take away all this friction from my life so I can focus on the things I love: my family and my hobbies. A personal agent will revolutionise my life in a way no other technology has.

I'll assert that 99% of people don't have any application for personal assistants, outside of work (perhaps). Most of us aren't so buried in personal obligations that the value of a half-assed "assistant" (which might violate our privacy) is worth the tradeoff in complexity or security.

The pitches for Dots, Grok Bot, etc., have been spectacularly uncompelling for me as a civilian. I don't need or want a bot to book my flights or read my inbox. I'm not so busy that these things have a value high enough to be worth the effort.

As a worker or an entrepreneur...maybe. Depends on the task, and how many other people were flinging slop at me on a daily basis. I could see the value of a calendar bot, or an expense reporting bot or a bullshit paperwork bot or a slop-return (slop jai alai?) bot, but there's no way in the world I'd give a bot access to the information necessary to, say, do my personal taxes.

I have been kind of disappointed by AI science-wise. I was anticipating more advancement. Experiments do take time, so I suppose they could be pending.

Why would you think this? Giving orders to software is such an ineffective, time consuming, error prone way to interact compared to using the interface yourself. That sounds delusional, and would be such a waste of resource, horrendous engineering and user experience

> if you haven't used the agentic tools with a frontier model in the past month or so

We keep hearing versions of this same drivel since late 2025. This time it's different! Now it's really super duper amazingly awesome powerful and if you don't believe it, just trust me bro.

several of the top replies on the x thread are "here's your post as a picture|video generated by AI"

Not why they are useful. Useful is load-bearing word (hehe) here, it spans from assisting someone by 1% or by 99%, very vague and not concrete.

I was talking about omitting mentions why LLMs usefulness is efficient - in coding specifically and STEM in general to a lesser degree.

And this is really a crucial thing, which do need to be brought up when discussing future forecast about multiple industries at once.

And the reason why LLMs are so efficiently useful for coding is because LLMs perform much better if any of the two conditions are met:

1) The problem at hand is formalized and transformed into a specification basically.

2) There is a formalized verification of the output done by external pipeline of different tools.

Any of the two drastically improve LLM performance, and coding activity uniquely has both of them.

Having a formalized task narrows down the scope of generation for LLM, and formal verification allows LLMs to iteratively converge on the verification pass check. Same with those math problems, they have a formal verification and LLMs can be prevented from wandering in a generated forest too much. Same in chemistry or engineering for some degree.

But other human activities usually have rather badly formalized inputs and no formal verification of the results. Karpathy is making a logical mistake (or deliberate misleading) of projecting a very unique industry (coding) on every other human activity without any nuance or analysis. He is biased, but due to his reputation and power he doesn't get called out on that mistake.

All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.

If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.

But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.

it also turns out that a non-ambiguous spec, is actually indistinguishable from code

I'm sorry but if you can finish spec'ing via 10 minutes of interview style questions, you're building to-do apps or a bloons TD clone not the next thing that replaces search engines.

yes, 1 billion in india can't wait to pay 100x their monthly wages to a chatgpt subscription, maybe it will help them make the recycled flip flips better

Try doing some of that blue-collar work yourself and enlist help of SOTA models for the task. I think it's not that far off. Your AI plumber may not look like a humanoid, and may not have the same hands that can "feel" the pipe and ears that can "hear", but they can hit the pipe with precise amount of force, and run Python in their head to analyze spectrogram, and learn much more from that than you could with your natural senses.

You can test this premise today. I already do, doing repairs around the house; the ability to send thermal cam photos or spectrograms (or just raw recordings) to Claude turns my phone into a tricorder with extra steps.

> understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge

I agree the data trove for these is not nearly as comprehensive as “the entire internet” was for LLMs. But LLMs paved the way - people are building the datasets rapidly now. I suspect this will look a lot like LLM rollout looked over the past 5-ish years. Unusable garbage… less unusable garbage… then suddenly, a fully capable model like Opus 5.5 or the like.

And the “problem” with the way the foundational models seem to work is that they don’t just acquire individual skills one at a time like we do - they go from knowing nothing to knowing everything.

> I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this

This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.

> I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.

Well supposedly some of the richer folks are busy building doomsday bunkers and whatnot, so I guess they aren't entirely blind to "political consequences"? Not sure what they hope accomplish there, unless they want to be king of some post-apocalyptic wasteland.

I for one am worried about our future, and have neither the money for a doomsday bunker nor the desire to live in the wasteland.

Yeah and this is what makes my worldview bleak atm. We as developers got these cool new tools which enable us to do our job 80% faster, but the end result wasn't less work. It was massive layoffs and making the rest of us work even harder to be "at the top".

When I was young there were already ideas about 4 day work weeks, it's now been 20 years later and so many things are automated, where are the shorter work weeks? I thought the point as a society was to not work ourselves off and actually enjoy day to day life while automation handles the boring part.

[dead]

This is the first time I hear this idiom and it's immediately shot to the top of my list...

It’s a good question - my sense is it is a combination of factors. First, robotics hardware is not at the $20k price point I hypothesized, and the durability is also not good enough to realize a payoff. Second, enabling remote operation would require significant additional investment in sensors and probably enhanced control workstations on the operator end - unlikely to be doable from a laptop. A reliable high speed internet connection would also be required on both ends. And third, human operators can be kind of a wild card. They require training, management, and carry their own security and safety risks.

Again, I’m not saying we’re there, it just feels like the control software is the main bottleneck, and the development of that is now being accelerated by LLMs.

Because context, at least in my experience, doesn't seem to encapsulate the sort of dynamism that I'm thinking of. Conversations are flat and repetitive. It still feels like a very good semantic search more than talking to an intelligent human.

I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.

Because we don't let them learn. Everyone runs the same static generalist models, instead of having every model instance learn from everything it does. New models come out occasionally, but they have not been trained on the private data you have not shared with the AI company.

You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.

Best I can do is a twitter post.

this is an entirely different mechanism - these people were taught everything is being hyped up and thus assume (usually rightly so) that everything is hype. what they're missing is the wisdom to not make this part of their individual and group identity, since that makes it very, very hard to update beliefs when facing overwhelming evidence.

I would question the narrative that humans lack the capacity to verify the output and would instead argue the people lack the incentive to verify the output.

The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.

For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.

Humans are not immortal and cannot spend all their time into review (especially unpaid). Even today, there’s so much knowledge around that you have to be specialist of a narrow domain to get to the frontier. Even in computing which is just approaching a century of existence.

Sol 6.1 is really good too though

Not sure if that's a fair criticism dang? I sincerely wasn't trying to be snarky or curmudgeonly, but I did want to make the point clearly that none of the benefits of AI have materialised in the eyes of the average person, only the downstream costs. To my eyes, that's a salient, reasonable, and wholly representative position to share.

Did the writing also contain emotion? Yes. As all good persuasive writing should.

It honestly escapes me why you would judge the juxtaposition between everyday life of everyday people and a very specialised minority as snarky or curmudgeonly. Was there too much emotional impact?

There are so many comments on HN that would justify your attention with regard to this specific rule. This is not one of them.

I think they made a really good point about this and I wouldn’t really call it snark.

How is this comment worth a moderator warning? Does not make sense to me, the comment is fine.

Maybe I don't understand the word snarky but this seemed fine.

What sets this particular comment apart from so many others on this thread that you felt the need to comment on it specifically?

Can you point out what a good way of expressing disagreement or skepticism about a subject matter looks like on HN?

[flagged]

And the LLMs exist to further those problems. People can't afford houses? Well, now they won't be able to own computers either. That's a feature, not a side effect. Once the K-shaped economy progresses enough, you won't even be allowed to use the LLMs. They'll exist only to be a tool for the new feudal class who owns you.

Unless you destroy their wealth now.

> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.

Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.

Yes, many of the problems are 'features not bugs'. There's no reason to think the AIs won't dream up more of these'features' too.

What I like to call at work "this is not a technical problem". There are many problems we could easily solve if we had the right structural incentives. But instead we have politics, processes, layers of management, etc. All that doesn't disappear (yet?) with AI.

> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.

When I asked why Musk doesn't put his money towards non-profits my Musk supporting friends say it's because he "doesn't believe anything but for-profit entities can solve the world's problems".

So why isn't he solving homelessness, famines, war, etc. with capitalism?

To add insult to injury, we're being told: "Just trust us bro, however we make things worse now will totes be redeemed as the prophesied god-mind magically fixes it later."

Strangely, they never discuss the possibility that Multivac might seize all their wealth and cast them down into average-ness. What if it actually comes about, but then it tells us to do the things we already refuse to do?

Agree. I’d only add that I think there is incentive for them to be solved, but there are greater incentives worth more to those leading the charge.

The job market had changed that's for sure.

That's what I'm saying.

Not only from the big companies. Where is the explosion of disruption ?

Sure, it did some meaningful things:

* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production

* Less junior hiring and suppressed wages

* Culture of uncertainty and fear

* Computers are really expensive now

* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health

It has greatly changed for the worse I can't get all the flashing lights out of my face now, more algorithms to hook you in, more social media, more gambling, more news, more things to catch your attention.

More noise. The world is just so noisy now.

Even in software, there are many things that haven't changed.

e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?

https://github.com/rust-lang/rust/issues

https://github.com/python/cpython/issues

What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.

So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?

---

Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.

(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)

>You could look around in 2016 and 2026 and honestly not much has changed

I find 2026 much more repulsive compared to 2016.

Mathematics and programming is verifiable and fully controllable in a digital environment.

Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.

Look at all the roadblocks for data centre buildouts which the industry certainly has a financial incentive to complete as fast as possible.

OpenAI and Sam Altman are literally backing a nuclear fusion power startup.

Electricity infra is not a technical problem. It’s been solved for decades. It’s a political problem, and both sides of the isle have had plenty to do with getting in the way

A lot of it isn't low friction really just in terms of political will it is high friction. E.g. housing issues are due to failing to upzone to meet demand appropriately. Solving housing is therefore free. It could be done today in any city in a housing crisis with the stroke of a pen, and before long, the market will equilibriate at a level better afforded by the median wage earner in the economy. The decision not to do this is a deliberate one: prices increasing is very much the goal of this sort of policymaking.

Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.

The priority is profit, not people.

Soon: The super-intelligent AI is telling us to cover parking lots and other flat roofs in solar panels.

It's kind of weird to say it's "taken off" recently. Yes what we have today is far better than half a year ago and solving new real use cases, but that was far better with more uses than we had half a year before that, etc, for years now. On their best usecases LLMs were already enormously useful years ago, and it's still very spiky.

As a user of software I see no improvements. What I see is that I need to download software more often, it brings more features I don't need and it has been as buggy as ever (if not more). And I can say that about Operating System I use, social networks, games, etc.

It took a long time for internet to transform society. Its still not fully baked in. These models that we have right now are transformational.

In other words, it's taken a highly enjoyable and lucrative vocation from vast swathes of workers - a lot of which make up this sites audience! - and instead concentrated more productive power in the hands of a few megacorps.

You gave your own example for the doom and gloom.

[dead]

What time scales are you using to make this judgement?

Everything isn't getting worse constantly. The data is quite clearly the opposite. How old are you?

It's not true that everything is more expensive. Nominal prices are higher, but incomes have risen faster than inflation. Purchasing power of the average person in 2026 is higher than it was in 2016.

As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.

Terence Tao and the apparently newly formed Association For Human Mathematics. The post is online-famous now for the line "Mathematicians did not ask for this work to be done."

https://terrytao.wordpress.com/2026/10/07/ahm-statement-on-o...

I'm curious, care to name a few things that have changed in Asia with AI (and wouldn't have without)?

> much has changed

Trust me bro

That is not how math works. There are reasons why the Riemann integral is pervasive and the Labague integral is rarely used unless needed. Just because you can do something does not mean that you should.

Do you blame them for getting impatient? They’re burning cash and quickly running to a point of never being able to justify it. The hardware they buy will be replaced with even more expensive hardware before they start to make a profit. It’s not sustainable and the market hasn’t figured that out yet. At least they’re starting to question it.

These backward looking analogies are tiresome.

Whether you like it or not - technology causes compression of life cycles. You have a hit product? Great - I expect you to have crazy high growth rate. That’s the expectation of a world that is so deeply interconnected.

Buy a dishwasher, you won't regret it!

> besides taking away my job

> self driving cars launched in my area

It sounds like the transformation would be a massive collapse of the economy with scores of people unemployed.

> Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.

Maybe, but more likely it will be the humans who control the AIs and hold the power who will make this decision.

Yes, and additionally, on the other side, there are a bunch of people using it to build exploitative products, spam, weapons, etc.

Almost every time I use an AI chatbot it eventually turns out it's lying to me. I would not trust them with any of the things you mention.

Personal agents are wonderful! You can ask for hiking trails and it will customize one for you, frequently resulting in emergency calls for rescues! Your purchasing/selling AI will arrange sales for you but forget to tell you, leaving you in breach of contracts you didn't know about! Your personal nutrition agent will absorb all the broscience and customize a diet that will result in hair loss, but at least your in ketosis, amiright! And instead of hiring a plumber/electrician/contractor to do work subject to code, you can just ask an AI to tell you how to do it and then wonder why your buyer is suing you for $$$!

(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)

[dead]

A decade? Better not tell the investors that!

China is building good AI by distilling American models. I think that's a great thing, but Chinese progress is currently dependent on American progress, because China lacks the compute to train the largest models (as stated by DeepSeek's founder[1])

[1] https://sumsec.me/2026/liang-wenfeng-investor-meeting-transc...

I was listening to a Verge podcast in which Nilay Patel remarked that of the AI companies started pushing for universal healthcare, AI would become a lot more popular. But no signs of this.

You are misunderstanding what mathematics research is. Its objective is the development of human understanding of mathematics. This requires open problems to research. AI are “solving” the open problems without creating any human understanding.

OpenAI to this day is not aware if they solved or not Navier Stokes.

Mathematicians are not gate keepers but I understand the frustration. If someone claims they spent 30 million dollars to solve one of the most difficult problems yet there’s no new knowledge, understanding, nor certainty. Then what’s the point besides marketing?

Uber was a massive mistake on every level.

This is what is so frustrating about this whole ai thing is we aren’t learning from the mistakes of even the very recent past

I agree with you, but those examples are going to fall on deaf ears. The pessimistic mindset will say that smartphones were a terrible invention that's ripping society apart, Uber is an unethical company exploiting workers, etc.

I would have expected apps like Amazon and websites like various Meta properties to be even slightly less broken and shitty.

Like, just at a bare minimum.

These are “use all the AI” companies with infinite resources, after all.

Sounds like collective psychosis.

Who indicted anyone?

My prediction is that while those “advancements” may come to cars, the only practical differences will be lower quality products, cost savings for the car companies, probably more expensive for consumers, and somehow it will also be collecting more data to sell to advertisers.

If you look at the average price of a new car, adjusted for inflation, it has pretty much always gone up aside from recessions. There have been plenty of advancements in car technology, they do not make them cheaper. It actually gives reason to charge more for them due to perceived increase in value.

Just a few more weeks, a few more hundreds of billions, it's right around the corner....

This only works if there remains a viable and attractive consumer market for these advanced goods. AI is both an imminent wealth shock and perhaps the largest structural change to the economy in history as it threatens to drive the value of human labor to zero over the next decades.

In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.

AI car navigation is here, but driving is a social act. Have you ever tried to keep an automated taxi out of a private driveway or the scene of an emergency? There is a lot of work to do and limited evidence of LLMs raising the bar.

Ah yes, the famous "trickle down" theory, which you may know from its smash hit "trickle down economics"! Surely this time it'll be different, right?

I think he means LLMs, we knew about RL already.

Worse, the view is largely phrased as gaslighting AI. Blaming the cause of particular grievances that coincidentally have been sourced at the same time as the rise of AI, is purposeful.

> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.

None of this has to do with AI.

> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)

Plain wrong on both counts.

> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.

Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.

As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes. There are still plenty of obstacles to overcome but the number of AI governance and steering groups that I have been apart of are indicative of what's happening in a major industry.

This is an interesting perspective.

Nevertheless, my counterpoint to

> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.

> They would be starting normal companies to disrupt the global economies of the world.

> Insurance companies. CAD companies. Compliance companies. Security auditing firms. Logistics software. Tax automation. ERP replacements. Drug discovery tooling. Industrial design. Legal ops. Accounting systems. Boring vertical SaaS with insane output per employee.

is:

Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.

Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.

Probably preaching to the choir here, but the software is just a symptom.

The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.

> Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.

Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.

Why and how will it be devastating specifically?

With AI software replication, I wonder if shareholder value even increases. Services accounts for 80% of American industry, what happens when big software companies loses revenue to piracy/recompilation i.e. Adobe, Unity, AutoDesk, tax/law software, etc..

And the human society was entirely destroyed, but for a brief moment we were able create share holder value.

[dead]

Exactly, we're all software engineers here and even for us it took weeks/months to get to a position where we can really leverage LLMs (think platform, security, observability, etc).

Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.

You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.

How much time do we have to wait before we are allowed to criticize AI? GitHub copilot has existed for half a decade. Cursor, Claude code, codex, have all existed for over a year and have seen more investment that practically any product in history and have almost 0 barrier to entry. Where is my decade of software progress in 1 year, because that’s what 10x productivity looks like! All I see are the same old broken products and services from basically the same handful of big players, higher costs, and more bugs. Remember when every year iOS and Android saw loads of new features and a radical rethink every few years? And that happened on ancient hardware and in a significantly smaller industry. Where is anything like that happening now?

> to just hear a friendly voice

This is far from being proven as a net gain and there are massive lawsuits incoming related to AI recommending suicide to kids and others.

All you said is true, but on the flip side it is giving an individual user huge power over software. To create their own, to reverse engineer proprietary ones. Decentralisation of such power can only be a good thing.

Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.

This is like saying food is shittier today because there are now more restaurants. No, food has never been better, there are just a ton of shitty restaurants.

"It is not from the benevolence of the butcher, the brewer, or the baker that we expect our dinner, but from their regard for their own interest"

When companies make profit, it doesn't mean we get screwed, nor even that they have a high profit margin. The grocery chain stores are a prime example. They typically make 2-3% net margins on revenue, and provide fresh, clean, reasonably well-tested groceries close to our home for that margin. Imagine trying to get equivalent quality fresh groceries ourselves from the producers, and I'm guessing it would cost at least three times as much.

Worst system, apart from the other ones.

Sure, but is that different than in the past? Those who could afford computers got good with computers. Those who could afford cars learned to drive. etc., etc.

Yeah I suspect Karpathy is too generous. I’m finding a lot of devs with access to the best models are not effective with them.

So I think there is are growing gaps even within the described groups.

That group does not exist. It's all hype, zero substance.

That's just cocktail party level of thinking. LLMs are getting better at math, code and logic (and marginally better at science and general knowledge) because these are domains that can be objectively verified and thus there is a potentially infinite supply of 'facts' to generate and train on. These are very powerful but ultimately very abstract domains. For everything else the messy real world and its physical bottlenecks gets in the way and there's little reason to expect progress to accelerate. It still takes months to get mice to reproduce and run experiments on, no matter how knowledgeable about biology the models have become.

If anything, in some domains frontier models have become worse - claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.

There was a time when people believed in Moore's law too. Now even Jensen Huang says Moore's law is dead. There's also the adage from investing: past performance does not guarantee future returns. A little CYA sure, but also acknowledgement of latent factors and the entropy of the universe and its affect on any predictive modelling one might do.

I’m not sure I agree that g explains all model capabilities. I think “claudespeak” is a consequence of the shift from training on RLHF to training on verifiable rewards. Thus the models seem to become worse at storytelling, fiction, readability, not better.

Even if we do accept that models have some “g” and that it correlates with all domains, there’s no reason to think that the current trajectory of LLM intelligence will grow more quickly or even as quickly. In every other technology we have an early period of slow growth during experimentation, and then a rapid period of growth, and then stagnation.

For all we know, 2026 could be the year of stagnation. It’s possible that we will reach finishing returns. If thats true, then it won’t matter how much that task correlates with overall g, because g won’t be increasing that much.

I don't agree that they get uniformly better at everything, especially subjective skills.

For example, they have become more and more unintelligible when you ask for explanations or descriptive text. They assume you see the same context as them and shortcut explanations.

I've had to craft a skill to get them to produce remotely understandable explanations of even mildly complex/non-mainstream subjects.

This may be Curse of Knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge on their part, and it also impacts human experts but still. Becoming better at one thing does not mean you become better at everything else, although I do agree with you that the better they become the more things there are they become good at, but their ability is still quite jagged and maybe increasingly so.

You are conflating intelligence with usefulness.

> Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing.

LOL. Human (and animal!) intelligence is notoriously jagged. We're all basically idiots except for very narrow areas where we focus.

Maybe, I'm at the point I don't know what to think anymore. But somehow, this feels still non-human. Humans don't need millions of hours of training and millions of samples to learn how do to something. We have proof that systems can deal with low-shot training. So why can't these? My point is when we have human level learning ability, then we're moving, if we expect expansion of capabilities to come from data alone, I'm prepared for disappointment.

I’m not sure. LLMs are superhuman in terms of workload (they are maybe also superhuman in ability too), but where does that lead us.. cheaper software?

That’s good for corporations who need custom software. Every large business has a massive backlog. In this area reducing the cost will massively increase demand.

But where’s the broader value?

Consumers and business already have too much choice in off the shelf software.

Ultimately most people just need word, excel and maybe 4 computer games a year.

> The most passionate, intelligent people are usually a decent indicator of where culture is going to go.

Many very smart people who I know are quite skeptical of LLMs and the hype around them. In my observation/echo chamber, the people who are very into LLMs are rather slick, career-minded people who love to present themselves as trendsetters for the "next fancy thing".

The most passionate, intelligent people 1. aren’t spending a lot of time on smartphones, video games and crypto and 2. aren’t as much smarter or better at predicting the future as the barely above average people who mistakenly count themselves in their ranks like to think.

Most of those technologies were controversial in their inception among the nerds of the time. Remember Eternal September? Linux/Windows people making fun of Apple hipster fans? Mainframe engineers deriding PCs as unserious and video games as childish?

>The most passionate, intelligent people are usually a decent indicator of where culture is going to go.

That's a naive view of cultural determinism. The software landscape would have been very different if a handful of lawsuits had gone one way or another. Hell, even Unix and Linux were hobby projects that accidentally got big.

Also crypto is a grift lol, one of those is definitely not like the others

Steve jobs wasn’t a nerd - he said he’s a hippie on the lost interview. He was also the first to bring to market colour screens, typography etc. He also defined the Internet as the defining ‘social moment’ way before anyone else.

So you’re wrong pal, sorry.

These have all been mistakes though.

You’re having a hard time distinguishing nerds and hippies.

Hippies live at the center of technology and humanities.

Nerds sit purely in technology.

> For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.

Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.

Its initial design was less than ideal for printability and it reworked the design correctly when asked to consider what it originally produced through the lens of printability.

Autodesk bundled an mcp server with fusion in the last few weeks and i expect things to get even better going forward.

[dead]

To add onto this, I feel like this has been happening a lot, where critics project backwards imagined claims or go nutpicking for random comments online to make something sound absurd and like everything is behind schedule and nothing is materializing, and then you can see other people (not necessarily the same people, the point isn't "look at these hypocrites") saying that it's all moving too fast. Both of these can't both be true in their strongest forms. I think what they have in common is that because they're relative measures to an unstated baseline, they're free criticisms you can throw at any development whatsoever, regardless of what is or is not turning up.

Also seems weird to me. Claude cowork could just do my monthly tax receipt collection and compare with my account better than the bookkeeper I pay for it. That is a very cross sectional problem

No, I’m talking about the recent DevDay.

Also, yes there are still bugs in Claude Code. I experience them nearly everyday.

It is markedly better than early days, but still not the best harness.

The best software written with agents seems to come from people outside of the labs (see pi, for instance — or all of cloudflare’s recent work)

Which makes me question either the model, or the holders …

I don't use either myself but I'm in a position where I get to see what it produces for other people. Here's a recent (2-3 months old) example: A zsh script that immediately invoked a Python interpreter that immediately invoked a shell command and iterated over each line of output looking for one of a few strings to match... So, `<cmd> | grep "(stringA|stringB|stringC)"`, but a whole lot dumber.

Thanks, but no thanks, I don't need that kind of code in anything I'm working with.

Claude code is an incredibly buggy mess. I have never used any other TUI that regularly has rendering errors or that is anywhere as laggy as Claude.

Claude Code still doesn't have a working scroll back buffer in their new renderer, it regularly flips its shit and mixes different pieces of history.

Claude Code still can't get reasonable performance without writing a "game renderer" (that doesn't work)

Claude Code is still written in JavaScript, eating hundreds of megabytes to make a shitty TUI whose literal sole role is to send API calls.

Claude Code is software made by amateurs.

Couldn't agree more. I find a new bug in the VSCode Codex extension every day... quantity != quality!

One solution to the aliens paradox is that they are so advanced they can hide from us. Maybe the killer software is kept inside the labs? I doubt it.

Be the change you want to see in the world: Attend or host an AGI society event to have that conversation:

agi-society.org