Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
I tested both myself and a weak bot against Astra xhigh, https://lichess.org/study/27lCQqDa. It's still pretty bad at chess, though it takes longer to devolve into illegal moves.
So give me a falsifiable point in time, a model you would claim succeeds at the task. One does not get to hotfix-patch updater out of the pressures of reality.
Today is the day.
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
By the promise of it, llms should be able to both adhere to grammars, or go free form where necessary. I mean, doing math is supposed to be strict but in practice it's a somewhat educated random walk in the space of correct lean theorems.
1. It’s hard to trust a 2026 paper that’s showing results for such old models.
2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.
3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
so prove it! get a public repo out there, have it play against some open source engines
also I think the operative letter in AGI is the G - and if the G is short for 'variably competent savant-like hyperfocus on certain kinds of software coding and not any other general skill' then its not really G at all, is it?
> People who are good at it rely more on experience and deep domain expertise
People are good are 1900 or 2100 above and the top ones who spend decades in the field i.e. deep expertise are well in the 2200-2700 range.
A 1100 player is none of these things, they are purely relying on strategic reasoning there is a good chance they cannot name a single opening or articulate clearly why a move was appropriate. 1100 is quite low bar.
The last post on HN I read was about someone using LLMs to reverse engineer an Apple GPU driver for linux in a month. The top comment points out how the poster must have had specialist internal domain specific contact with Apple. But then the thread concludes that wasn't the case and that this would take domain experts years to do.
> "current frontier models need laborious oversight and guardrails on even the simplest tasks"
I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.
What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?
Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?
> Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
It can be very interesting and even entertaining to know where models don't do well. I don't find something like chess to be very instructive about anything though, nobody is going to pay for AI to play chess at any significant scale even if it could do it perfectly.
> The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
I have found that not to be the case, and I am not an AI power user or cheerleader by any means. There's probably a bunch of even "simplest" tasks where AI doesn't do well and might never. That doesn't take away from the cases where it works well and is a productivity booster. It doesn't even have to be solving millennium prizes or any other breakthrough creativity or reasearch, it's still very useful in places.
This isn't how intelligence works. The LLM may not be able to play chess directly through inference, but it can write a program to do it and execute that program. Same as how human intelligence works. We can't fly, but we can build planes.
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
There’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.
Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves.
Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.
> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player.
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
I don't see why this is such a big deal. Nobody's using LLMs for chess, but even if they are, just give them Stockfish as part of their harness. They don't need to do everything themselves as long as they're intelligent enough to use tools.
The fact that they can play chess at all despite having no specific training for it blows my mind, and the fact it doesn’t do the same for many others shows just how far they’ve come and how fast.
Great article, but the lack of sentence capitalization makes it unnecessarily difficult to read.
Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...
Even assuming this is how the AI companies are being valued (they're not), the numbers are off.
The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.
Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.
> The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually.
What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).
> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that
So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.
It does not have to be 16% of all jobs, but 16% of any given job, i.e. AI stays in an augmentative role rather than a complete job automation. The simplistic analysis is if a tool makes you X% faster, that can be worth X% of your salary to your employer.
Unfortunately, I do fear that AI adoption will go beyond augmentation to automation, and I do fear an economic shock. Just posted this down-thread: https://news.ycombinator.com/item?id=49722616
You are comparing company valuations to annualized revenue (as approximated by some fraction of total knowledge worker compensation). Valuations are (roughly) based on the sum of all discounted future cash flows, not just the current year’s revenue.
When thinking about these valuations, shouldn’t we try to quantify how much knowledge work becomes obsolete if other knowledge workers are automated? I.e. there are a huge amount of knowledge workers employed in businesses that create tools for other knowledge workers. AI won’t automate their work, those businesses will just cease to exist.
And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?
I think you are committing the lump of labor fallacy [1]. Lots of jobs will disappear, but others will appear. Lots of things (both intellectual and material) that are produced nowadays by humans will be produced in the near future by AI. But humans will be needed to do new things.
Take the Hugging Face incident. Why did it happen? Because the people whose task was to set up a testing framework took shortcuts. Why did they? Because there weren't enough people who were assigned to do the job. Why not? Because the job is too new and not enough people are qualified to do it. It's a job that simply did not exist 3 years ago. But 3 years from now, this job might very well employ tens of thousands of high skill knowledge workers.
Oh for sure, this was a simplistic analysis assuming AI adoption caps out at some X% of job responsibilities where X << 100%.
Unfortunately, I fear that may not be the most likely outcome. I've posted some comments on this before, but when I start thinking about how deeply everything will change once people figure out how to properly leverage AI, I see no outcome other than significant, widespread job losses.
As you indicated, at that point we will have much a bigger problem than the valuation of the AI industry. I'm not sure how it will get solved, I just know it will HAVE to be, because it would be an existential problem for everybody: people, governments, even the billionaires! Because now consider the 3rd order effects: if nobody can buy the stuff that's produced, how can billionaires get even richer? ;-)
> It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.
You’re right; given that most of the money in the AI market is injected through OpenAI and Anthropic (which collect it through both selling equity and through customer revenue), the 7-8T is just a derivative of that.
"are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers, "
No, they're really not.
They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.
And that they will capture most of that ... which they won't.
The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.
The money is going to Nvidia, who have the most powerful position.
A bit like how a retailer can take all the margins of some innovative product, if they own the channel.
AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.
The challenge with estimating abilities, is that we don’t know what the models can achieve if we just burn enough money. The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.
It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.
But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.
Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?
The whole point of this post was that it's questionable what can be achieved without huge investments into oversight and steering, because navier-stokes was a topic with an unusual level of specification. The problem itself was a specification. Such situations are rare in real-world scenarios.
AI agents are good at solving well-specified tasks, not at solving problems. They do well in fields where the cost/effort of specification is already part of the business.
Short and to the point! Open and cheap models will undercut the big labs continuously. The blast radius won't be pretty once spending commitments knock the door.
Open models wont be open for long. No one is going to release an open model capable of chaining zero-days. Even the Chinese aren't that reckless because it will just be turned around and used against them.
Isn't that assuming that fix won't be implemented?
Zero days are valuable because they can be exploited but if the pace of exploitation is faster (which I'm not sure is the case), then the response WILL be faster, even if it means going offline. Institutions that won't will simply go offline by losing their data or becoming unprofitable due to ransomware.
Now for components that are core to the infrastructure, say OpenSSL, there is already a TON of attention and efforts, including red teaming, so it's not as if it's opening floodgates.
Sure low hanging fruits will get picked either faster or a at a larger scale, say a random outdated IoT device at your local flower shop, but for the rest, I don't think it's realistic to expect no response.
Security, digital or not, has always been an arm race. New threats means new responses specifically by incorporating the threat.
As compute prices fall it gets easier and easier to make "frontier" models. So it's inevitable that commodity, open source models of equivalent capacity to today's "frontier" models will be available to the public. Remember this is just weights, anyone can download them and run it whenever they like. The only constraint is compute.
depends on the blast radius of zero-days, it's not like there's a continuous immediate release process for these models; they can eval internally before releasing publicly
I think bearish on LLMs for automation, and bullish for LLM+human experts in specific fields, is about the right expectation for current architectures.
Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.
The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.
> This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
Any reason why that can't be solved through context management and keep-forward scaffolding?
> the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse
when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns
I don’t know who you’re talking about, even the most bearish people like Gary Marcus and Ed Zitron acknowledge that LLMs are useful in these same cases the OP admits. Gary Marcus is even still a long term AI advocate, he just doesn’t think LLMs are enough and we need more foundational breakthroughs. Zitron says it’s valuable technology but not worth the trillion dollar valuations the frontier labs are claiming.
The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.
Even a lot of the people who think that LLMs are a dead end think that we will soon find something signficantly more powerful, which I find deeply alarming. I don't want to know what my white-collar knowledge work will look like in a decade or 2.
Gary Marcus is an especially puzzling addition. If I recall correctly, he has made statements along the lines that superintelligence this century is more likely than not. If you’re AGI-pilled that might read as bearish, but that is still extremely rapid progress in the grand scheme of things.
I can see current limitations, but how do you expect capabilities to change in the next few years? A repeat of the gain that happened in the last two years feels like it would be significant, even if it took a little more than two years this time around.
Yea I get the bearishness from my own personal experience.
Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.
But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.
Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.
Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.
So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.
I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).
> those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed
came here to say exactly this. in fact, this is probably why we are not seeing a lot of AI application on customer service use case, and when we see one, it's almost always frustrating.
LLMs are basically multi-dimensional magic mirrors.
Depending on where you point them, they can be incredibly useful.
They can even be useful when you point them at each other (though increasingly difficult to get good results).
I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
I was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.
but maybe... the said human has also read every other piece of text ever written, including ones about other (similar?) board games, which in aggregate vastly dwarfs whatever he has read on chess, that non-chess reading could have corrupted whatever he's learned about chess?
Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.
being a bit more specific: the sample efficiency of humans is orders of magnitude larger for more abstract concepts. the same doesn't hold for memory-intensive tasks though (like any kind of trivia), but that only takes you so far.
> current frontier models need laborious oversight and guardrails on even the simplest task
As models advance, we shift the goalpost for what "simplest task" means. Before, "simplest task " meant "write a coherent English sentence." Now, "simplest task" means autonomously fix, review, and merge a bugfix.
I agree with the caveat that it's more like a cracked junior engineer who can manage swarms of interns.
Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers
That's a reason to be bearish about AI companies, not LLMs. But is it even true? OpenAI and Anthropic have each reported ~50 billion in revenue with ~900 billion valuations. That's a high ratio but I'm not sure if follows that the only way it pans out is if we get "fully automated drop-in replacement for most knowledge workers".
It wouldn't shock me to see those revenue numbers scaling up to where they need to be over the next decade ( to, say, ~400 billion) without ever achieving drop-in worker replacements.
I looked at the math and I think it's true. Remember revenue is just sales, not profit. These labs are shooting for > $1T valuations, which traditionally means your PROFIT is at least 1/20th or 1/30th of that (so let's say minimum 30B$/year PROFIT).
These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?
So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).
In particular, I found this very misleading or irrelevant:
a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of
The reason silicon design has such verification to design ratio is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months to fab a chip, and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
I don't think you can extrapolate these very industry-specific facts to judging LLMs.
> The reason ... is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months ... and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
Aren't you just describing waterfall? That's still very prevalent in software engineering, and pretty much any other type of engineering – civil, chemical, building, architecture, drug discovery.
It's typically true that software can fail faster and cheaper, but it's also true that the costs are still vastly higher to fix later in the process.
No. Silicon is on another level. Which is why the EDA verification is an industry on its own.
Sure, there are some software that have similar "can't have bugs" requirements. I imagine the computers on Moon missions also had that kind of high bar. I wouldn't use NASA requirements as a proof for how LLMs should be used.
Doesn't really even need to look like it. If you can verify rewards, RLVR will optimize really really well. If you can't... it's a struggle. There are probably fewer fields where you can verify rewards than one might hope.
> current frontier models need laborious oversight and guardrails on even the simplest tasks.
This is only true if you are concerned about the intermediate steps of the model as opposed to the outcome. The huggingface hack was a perfect example of the model doing whatever it takes to accomplish the goal of maximizing its score.
refreshing to see amongst the endless tide of "i haven't written a single piece of code since 2025, llms are so good that they have already replaced everyone" gaslighting
> the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.
the issue most of you seem to not realize is that when you put these models in a loop, you are able to do more and more insane and cool things.
have you guys actually designed, built, and deployed agentic workflows?
it is actually quite hard, requires tons of time spent on evals and testing to ensure accuracy, but when it starts to work it is mind blowing.
there is no going back.
listening to people yap about AI when they have only surface level or one dimensional exposure to LLMs and "AI", but have not actually put innovations to work IN PRACTICE.. is a waste of time
great, autonomous LLMs will fail. that's actually perfect. they work amazingly well when we're telling them what to do. no autonomy needed, no destruction of humanity. that's all win
DO you know those ice cream shops that sell 30 different flavours. Everybody likes a different flavour, some people dont even like ice cream and buy nothing.
Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.
Summarise means different things to different people.
This April 2026 paper is a fun and related read.
https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
> current frontier models
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
I tested both myself and a weak bot against Astra xhigh, https://lichess.org/study/27lCQqDa. It's still pretty bad at chess, though it takes longer to devolve into illegal moves.
So give me a falsifiable point in time, a model you would claim succeeds at the task. One does not get to hotfix-patch updater out of the pressures of reality. Today is the day.
They still need supervision though
The actual current frontier plays somewhere around GM level.
https://chessbench-ai.github.io/#leaderboard
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
49 replies →
[dead]
Llm systems are not really build for adhering to a grammar (other than "a string og tokens").
It is also not clear whether the llm adhering to a grammar is necessary for intelligent agents.
Certainly,a harness can easily correct for it.
It seems absolutely crazy to me to expect an LLM to code a solution to a problem while also not expecting it to be able to adhere to a grammar.
1 reply →
By the promise of it, llms should be able to both adhere to grammars, or go free form where necessary. I mean, doing math is supposed to be strict but in practice it's a somewhat educated random walk in the space of correct lean theorems.
Harnesses do correct things, sure.
1 reply →
1. It’s hard to trust a 2026 paper that’s showing results for such old models.
2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.
3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
Good science, properly digested and presented takes time.
The idea that anything other than a breathless blog post about the latest model snapshot is useless is really poisonous to proper debate on AI issues
so prove it! get a public repo out there, have it play against some open source engines
also I think the operative letter in AGI is the G - and if the G is short for 'variably competent savant-like hyperfocus on certain kinds of software coding and not any other general skill' then its not really G at all, is it?
4 replies →
> Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
A bash script can clone and build stockfish, feed in human moves, and reply. By your standard, this bash script would "destroy any human at chess."
Are you interested in assessing the intelligence of the model, or the intelligence of the tools the model can use?
1 reply →
> People who are good at it rely more on experience and deep domain expertise
People are good are 1900 or 2100 above and the top ones who spend decades in the field i.e. deep expertise are well in the 2200-2700 range.
A 1100 player is none of these things, they are purely relying on strategic reasoning there is a good chance they cannot name a single opening or articulate clearly why a move was appropriate. 1100 is quite low bar.
3 replies →
> Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
Delusional, but then Claude fable also isn’t beating any human at chess, the engine is.
The last post on HN I read was about someone using LLMs to reverse engineer an Apple GPU driver for linux in a month. The top comment points out how the poster must have had specialist internal domain specific contact with Apple. But then the thread concludes that wasn't the case and that this would take domain experts years to do.
> "current frontier models need laborious oversight and guardrails on even the simplest tasks"
I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.
What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?
Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?
> Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
It can be very interesting and even entertaining to know where models don't do well. I don't find something like chess to be very instructive about anything though, nobody is going to pay for AI to play chess at any significant scale even if it could do it perfectly.
> The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
I have found that not to be the case, and I am not an AI power user or cheerleader by any means. There's probably a bunch of even "simplest" tasks where AI doesn't do well and might never. That doesn't take away from the cases where it works well and is a productivity booster. It doesn't even have to be solving millennium prizes or any other breakthrough creativity or reasearch, it's still very useful in places.
why cant models make a tool call to stockfish? its like saying model can't execute python for complex math calculations
Because then it’s not playing chess, stockfish is?
Exactly. All these nerds saying cars make bad submarines. Well duh.
I wonder how current models would fare. The ones they tested are fairly old now.
This isn't how intelligence works. The LLM may not be able to play chess directly through inference, but it can write a program to do it and execute that program. Same as how human intelligence works. We can't fly, but we can build planes.
Human beings can play chess directly without coding up a tool.
5 replies →
Thanks for saying this, feels like everyone has gone insane over this stuff.
6 replies →
The story isn't so clear cut.
The caveat is: It depends on the task.
Are there reams of chess moves that the model can train off of? No.
Are there reams of math papers the model can train off of? Yes.
> The caveat is: It depends on the task.
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
There’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.
> Are there reams of chess moves that the model can train off of? No.
This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.
3 replies →
>Are there reams of chess moves that the model can train off of? No.
For real??
[dead]
[flagged]
Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves. Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.
> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player.
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
19 replies →
I don't see why this is such a big deal. Nobody's using LLMs for chess, but even if they are, just give them Stockfish as part of their harness. They don't need to do everything themselves as long as they're intelligent enough to use tools.
The fact that they can play chess at all despite having no specific training for it blows my mind, and the fact it doesn’t do the same for many others shows just how far they’ve come and how fast.
Great article, but the lack of sentence capitalization makes it unnecessarily difficult to read.
Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.
Not perfect but hope it helps.
Gosh, getting downvoted for trying to help a stranger, what is wrong with you people?
The premise in the very first point seems off:
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...
Even assuming this is how the AI companies are being valued (they're not), the numbers are off.
The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.
Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.
> The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually.
What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).
> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that
So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.
It does not have to be 16% of all jobs, but 16% of any given job, i.e. AI stays in an augmentative role rather than a complete job automation. The simplistic analysis is if a tool makes you X% faster, that can be worth X% of your salary to your employer.
Unfortunately, I do fear that AI adoption will go beyond augmentation to automation, and I do fear an economic shock. Just posted this down-thread: https://news.ycombinator.com/item?id=49722616
You are comparing company valuations to annualized revenue (as approximated by some fraction of total knowledge worker compensation). Valuations are (roughly) based on the sum of all discounted future cash flows, not just the current year’s revenue.
When thinking about these valuations, shouldn’t we try to quantify how much knowledge work becomes obsolete if other knowledge workers are automated? I.e. there are a huge amount of knowledge workers employed in businesses that create tools for other knowledge workers. AI won’t automate their work, those businesses will just cease to exist.
And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?
I think you are committing the lump of labor fallacy [1]. Lots of jobs will disappear, but others will appear. Lots of things (both intellectual and material) that are produced nowadays by humans will be produced in the near future by AI. But humans will be needed to do new things.
Take the Hugging Face incident. Why did it happen? Because the people whose task was to set up a testing framework took shortcuts. Why did they? Because there weren't enough people who were assigned to do the job. Why not? Because the job is too new and not enough people are qualified to do it. It's a job that simply did not exist 3 years ago. But 3 years from now, this job might very well employ tens of thousands of high skill knowledge workers.
[1] https://en.wikipedia.org/wiki/Lump_of_labour_fallacy
Oh for sure, this was a simplistic analysis assuming AI adoption caps out at some X% of job responsibilities where X << 100%.
Unfortunately, I fear that may not be the most likely outcome. I've posted some comments on this before, but when I start thinking about how deeply everything will change once people figure out how to properly leverage AI, I see no outcome other than significant, widespread job losses.
As you indicated, at that point we will have much a bigger problem than the valuation of the AI industry. I'm not sure how it will get solved, I just know it will HAVE to be, because it would be an existential problem for everybody: people, governments, even the billionaires! Because now consider the 3rd order effects: if nobody can buy the stuff that's produced, how can billionaires get even richer? ;-)
> It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.
You’re right; given that most of the money in the AI market is injected through OpenAI and Anthropic (which collect it through both selling equity and through customer revenue), the 7-8T is just a derivative of that.
"are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers, "
No, they're really not.
They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.
And that they will capture most of that ... which they won't.
The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.
The money is going to Nvidia, who have the most powerful position.
A bit like how a retailer can take all the margins of some innovative product, if they own the channel.
AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.
ai has a >10% chance of causing human extinction, according to anthropic big heads.
if that's true, you are wrong.
if that's false, anthropic is dishonest. why trust a dishonest company to be worth anything?
I think that the AI people believe in their own nonsense a bit.
Like - the guy on TV talking about 'AI will destroy everything' ... I don't think he's lying.
I think they are like we here on HN and Reddit and a bit caught up in our own thoughts.
If AI were unleashed, in raw form today, it could cause havoc.
Bad. Maybe very bad but I think we'd get over it.
It would probably trigger a recession (because we are in a bubble - it would pop it), and people would 'blame the AI' for sure.
But it would be a bit dot-com ish kind of recession.
The amplifiers would be geopolitical instability.
2 replies →
The challenge with estimating abilities, is that we don’t know what the models can achieve if we just burn enough money. The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.
It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.
But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.
Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?
The whole point of this post was that it's questionable what can be achieved without huge investments into oversight and steering, because navier-stokes was a topic with an unusual level of specification. The problem itself was a specification. Such situations are rare in real-world scenarios.
AI agents are good at solving well-specified tasks, not at solving problems. They do well in fields where the cost/effort of specification is already part of the business.
Short and to the point! Open and cheap models will undercut the big labs continuously. The blast radius won't be pretty once spending commitments knock the door.
Open models wont be open for long. No one is going to release an open model capable of chaining zero-days. Even the Chinese aren't that reckless because it will just be turned around and used against them.
Isn't that assuming that fix won't be implemented?
Zero days are valuable because they can be exploited but if the pace of exploitation is faster (which I'm not sure is the case), then the response WILL be faster, even if it means going offline. Institutions that won't will simply go offline by losing their data or becoming unprofitable due to ransomware.
Now for components that are core to the infrastructure, say OpenSSL, there is already a TON of attention and efforts, including red teaming, so it's not as if it's opening floodgates.
Sure low hanging fruits will get picked either faster or a at a larger scale, say a random outdated IoT device at your local flower shop, but for the rest, I don't think it's realistic to expect no response.
Security, digital or not, has always been an arm race. New threats means new responses specifically by incorporating the threat.
As compute prices fall it gets easier and easier to make "frontier" models. So it's inevitable that commodity, open source models of equivalent capacity to today's "frontier" models will be available to the public. Remember this is just weights, anyone can download them and run it whenever they like. The only constraint is compute.
depends on the blast radius of zero-days, it's not like there's a continuous immediate release process for these models; they can eval internally before releasing publicly
I haven't heard of the term "chaining zero-days". Now as a SRE I wont sleep.
I agree with you but I'm still worried about the safety of open weight models as well. Both aligned and unaligned models.
You said it like labs like open AI doesn’t know and don’t constantly make moves to prevent that undercutting
I think bearish on LLMs for automation, and bullish for LLM+human experts in specific fields, is about the right expectation for current architectures.
Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.
The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.
> This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
Any reason why that can't be solved through context management and keep-forward scaffolding?
Write the same sentence you just wrote back to me, but in only four words and let’s see if it has the same meaning.
2 replies →
> the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse
when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns
I really appreciate seeing a tempered take that's not literally denialist about current capabilities.
I don’t know who you’re talking about, even the most bearish people like Gary Marcus and Ed Zitron acknowledge that LLMs are useful in these same cases the OP admits. Gary Marcus is even still a long term AI advocate, he just doesn’t think LLMs are enough and we need more foundational breakthroughs. Zitron says it’s valuable technology but not worth the trillion dollar valuations the frontier labs are claiming.
The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.
Even a lot of the people who think that LLMs are a dead end think that we will soon find something signficantly more powerful, which I find deeply alarming. I don't want to know what my white-collar knowledge work will look like in a decade or 2.
Gary Marcus is an especially puzzling addition. If I recall correctly, he has made statements along the lines that superintelligence this century is more likely than not. If you’re AGI-pilled that might read as bearish, but that is still extremely rapid progress in the grand scheme of things.
> current frontier models need laborious oversight and guardrails on even the simplest tasks
It is literally denialist about current capabilities
why don't anthropic and openai ship yolo mode by default?
4 replies →
Thanks :) I do enjoy and use these things every day and the current capabilities are indeed amazing, just ludicrously overpriced at the frontier.
I can see current limitations, but how do you expect capabilities to change in the next few years? A repeat of the gain that happened in the last two years feels like it would be significant, even if it took a little more than two years this time around.
Yea I get the bearishness from my own personal experience.
Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.
But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.
Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.
Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.
So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.
I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).
[1] https://en.wikipedia.org/wiki/Wicked_problem
> those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed
Depends on what customer support means.
Typically it means knowledge retrieval from a KB or manipulating a control surface not visible to you.
came here to say exactly this. in fact, this is probably why we are not seeing a lot of AI application on customer service use case, and when we see one, it's almost always frustrating.
LLMs are basically multi-dimensional magic mirrors.
Depending on where you point them, they can be incredibly useful.
They can even be useful when you point them at each other (though increasingly difficult to get good results).
I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
I was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.
It seems like an LLM potentially could learn that way if each practice game it participated in was added to its training data.
3 replies →
but maybe... the said human has also read every other piece of text ever written, including ones about other (similar?) board games, which in aggregate vastly dwarfs whatever he has read on chess, that non-chess reading could have corrupted whatever he's learned about chess?
Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.
> Is this really any different to how humans learn
yes.
being a bit more specific: the sample efficiency of humans is orders of magnitude larger for more abstract concepts. the same doesn't hold for memory-intensive tasks though (like any kind of trivia), but that only takes you so far.
1 reply →
Also doesn't the very good ARC AGI 2 score of GPT-6 Astra kinda contradict this, since each problem is its own game with very different rules
> current frontier models need laborious oversight and guardrails on even the simplest task
As models advance, we shift the goalpost for what "simplest task" means. Before, "simplest task " meant "write a coherent English sentence." Now, "simplest task" means autonomously fix, review, and merge a bugfix.
Eliza wrote coherent English sentences.
I agree with the caveat that it's more like a cracked junior engineer who can manage swarms of interns.
Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers
That's a reason to be bearish about AI companies, not LLMs. But is it even true? OpenAI and Anthropic have each reported ~50 billion in revenue with ~900 billion valuations. That's a high ratio but I'm not sure if follows that the only way it pans out is if we get "fully automated drop-in replacement for most knowledge workers".
It wouldn't shock me to see those revenue numbers scaling up to where they need to be over the next decade ( to, say, ~400 billion) without ever achieving drop-in worker replacements.
I looked at the math and I think it's true. Remember revenue is just sales, not profit. These labs are shooting for > $1T valuations, which traditionally means your PROFIT is at least 1/20th or 1/30th of that (so let's say minimum 30B$/year PROFIT).
These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?
So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).
Not convinced by those points.
In particular, I found this very misleading or irrelevant:
a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of
The reason silicon design has such verification to design ratio is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months to fab a chip, and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
I don't think you can extrapolate these very industry-specific facts to judging LLMs.
> The reason ... is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months ... and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
Aren't you just describing waterfall? That's still very prevalent in software engineering, and pretty much any other type of engineering – civil, chemical, building, architecture, drug discovery.
It's typically true that software can fail faster and cheaper, but it's also true that the costs are still vastly higher to fix later in the process.
No. Silicon is on another level. Which is why the EDA verification is an industry on its own.
Sure, there are some software that have similar "can't have bugs" requirements. I imagine the computers on Moon missions also had that kind of high bar. I wouldn't use NASA requirements as a proof for how LLMs should be used.
Specifically: bearish on LLMs generally, not bearish on LLMs for pure math.
yes, huge for pure math and activities that look like it.
Doesn't really even need to look like it. If you can verify rewards, RLVR will optimize really really well. If you can't... it's a struggle. There are probably fewer fields where you can verify rewards than one might hope.
1 reply →
> current frontier models need laborious oversight and guardrails on even the simplest tasks.
This is only true if you are concerned about the intermediate steps of the model as opposed to the outcome. The huggingface hack was a perfect example of the model doing whatever it takes to accomplish the goal of maximizing its score.
So, if you are concerned about what the model does? Yeah.
I am bullish on AI. At some point well see some true advancements.
good post/take.
refreshing to see amongst the endless tide of "i haven't written a single piece of code since 2025, llms are so good that they have already replaced everyone" gaslighting
- i have bearish from day 1
- i have no idea how anyone thinks the mighty next token predictor is going to eradicate diseases and eliminate poverty https://blog.florianherrengt.com/how-llms-work.html
- i also have no idea what everyone and their momma on HN is running for more than 5 mins in the name of "agentic AI"
Won’t this change though?
> the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.
the issue most of you seem to not realize is that when you put these models in a loop, you are able to do more and more insane and cool things.
have you guys actually designed, built, and deployed agentic workflows?
it is actually quite hard, requires tons of time spent on evals and testing to ensure accuracy, but when it starts to work it is mind blowing.
there is no going back.
listening to people yap about AI when they have only surface level or one dimensional exposure to LLMs and "AI", but have not actually put innovations to work IN PRACTICE.. is a waste of time
great, autonomous LLMs will fail. that's actually perfect. they work amazingly well when we're telling them what to do. no autonomy needed, no destruction of humanity. that's all win
archive link in case i get hugged lol https://archive.ph/Z4gxF
[flagged]
[flagged]
[dead]
doomers gonna doom
All website should come with a Summerize button.
DO you know those ice cream shops that sell 30 different flavours. Everybody likes a different flavour, some people dont even like ice cream and buy nothing.
Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.
Summarise means different things to different people.
Thats a browser level task.