Comment by a_bonobo
3 months ago
>On my visits to the Bay Area, I would ask AI researchers or interns why they are doing their current research or projects, when in a year or three agentic LLMs could probably do them;
This is such a weird point to make that doesn't become correct just because everyone makes it, all the time. Why clean the ocean if some magic future tech will clean them? Why save the world now if some benevolent AI is 'just around the corner' and will do it for us? And people have been making this point for years now, and it's not like my job got any easier. I just got more AI.
https://www.poetryfoundation.org/poems/51294/waiting-for-the...
And I say that as someone who uses Claude Code in complex environments almost hourly; I, as the human, still have to do the thinking as Claude still 'can't jump' [1] and I have seen no evidence that they (or similar AI, any time soon) will 'jump' like a human brain does.
> This is such a weird point to make
I think it is a great point to make, because if everyone really believed that AIs will do everything without human intervention in a handful of years, as the marketing repeats again and again (AGI, singularity, etc.) and have been saying for years... why then get bothered?
Because we DO know LLMs have their hallucinations, limitations, perform tasks not previously seen way worse than humans, etc. And it seems that, for now, there is not a good or magic solution to it, it is inherent limitations of the paradigm.
Yes, you can feed more and more and more (curated data) and eventually make AIs excellent at task X or Y, but then you spend your time specializing those engines. So the work does not really disappear, it just shifts and you make it more replicable for a bound set of problems.
Needless to say that at some point I prefer to learn (and combine with AIs, it is ok) than acritically getting inputs from something until I become totally useless.
Unless we have a paradigm for which a fully autonomous AI can do everything, this will just become improving our productivity in some ways, with all the in-between bottlenecks that it has.
It's a very silly point to make to AI researchers specifically. If they don't work on those projects, the AI won't advance and won't magically be able to replicate the work in "one to three years".
Can you imagine scenarios that would make it less silly? I will give an example:
- The AI researcher might be working for a lab or company with much less funds than the top dogs. Are they likely to discover something that is worth it before a bigger model becomes more capable?
2 replies →
Relevant xkcd: https://xkcd.com/989/
Why spend money and time making the new flagship model when a future flagship model can make you the flagship model?
Then why not stop researching and doing the definitve model that will solve every problem? Why some people are not doing it?
Bc they are aware of the marketing and limitations. If they did believe it, then they would switch area of research.
> you can feed more and more and more
I have a meta thought..
Hypothetically what happens once there is no more data to be fed to the system? Are we expecting AI to invent its own data and reach full cognition?
Currently we are feeding it the data that humans created but if we stop (i.e "why bother?") thinking that AI will do it all?
This is a well-known problem that has been an issue for years now. You can't use models to generate data for models because it leads to "model collapse" where it amplifies quirks in the generated data until it's all quirks. Here is a random university press release about it (grain of salt etc)
https://www.utoronto.ca/news/training-ai-machine-generated-t...
In practice you can do it a bit (generated data from a better / different model is fine, some generated data might be useful if there is non generated data etc.)
1 reply →
The AI researchers are not the ones making the marketing, much less believing in it.
I agree. But this is not what you see on the headlines and what money-incentivized stakeholders are saying.
Is this really true? At least one very headline AI researcher is pretty much an Anthropic spokesperson.
1 reply →
> inherent limitations of the paradigm
This is such a weird point to make. We are currently ( only ) discovering that paradigm; we are not inventing anything. We found a bunch of laws that produce rather cool results but our paradigm is incomplete which leads more or less wordy or frame-rich weird stuff like hallucinations, singularity and so on ... it's childish, really and on that funny pseudo-profound, pseudo-intellectual, pseudo-spiritual ( personal opinion, if it gets you horny, you go, baby ) "universe consciousness unity, Rick James, bitch" level ...
Our bodies and minds need proper AI, not all the stuff we already outsource to middle and/or passionate men and women. Other species on the planet would certainly like to see us get augmented by AI so we can solve as many survivability issues as possible to keep as many ecosystems running long enough ... whatever that means but whether animals and plants are aware of chance and potential is another philosophical debate.
To individuals, software is a hammer and chisel, a knife, a brush and canvas, pen and paper, a reading help, and to a good amount of people it's a microscope and a fine scalpel.
To collectives, it's a tool to work on consensus and conventions, to share and gather.
It's baby steps for civilizations and it looks like our particular species is gonna get stuck in a puddle of our own monkey shit, with bottles of champagne in our hands and monkeys grinding up and down the few ivory towers in proximity.
> why then get bothered
Humans are on different levels. Most have decided that "nature realized the/a bug and wanted someone dead" or "their survival is a matter of chance" is not acceptable at all and some people decided that sabotage, poison, abuse, rape, murder are acceptable means to get chicken shit ...
The "paradigm" of life is far from explored/discovered, so we simply can't content ourselves with presumptions about inherent limitations of the LLM and AI paradigm for any other reason than to uncover ( not invent ) other parts of the paradigm.
We are happy with what AI can do for us but "AIs will do everything without human intervention" sounds weird because babies are born and the older they get and the less sabotaged ( vs influence, cultural manipulation ) they get to grow up, the more breadth and depth humans want to experience. For this they need to learn and use their hands & fingers. They need to feed body and mind to find what triggers what, and what excitement and curiosity are inherent and which can or need to be added/acquired/experienced extrinsically.
How many associations will we be able to make if AIs will do everything without human intervention?
No we don't know your 'points'
The Hallucinations are becoming less, significantly by now.
It also might be already were it is cheaper for one of the big few companies to spend millions and billions to teach the LLM / creating the training data necessary for an LLM to do something which it is not yet good enough due to the fact, that they sell this capability then to everyone who wants to use this capabilitiy.
We have not seen the end of Reinforcement Learning, which does need a lot less training data but more compute.
I'm 'vibing' on the side a handfull of small things, no LLM trained on particular what i'm asking to do. Its very capable of stringing together enough things so it can clearly follow handwavy things i tell it to do, analyse error messages, analysing screenshots etc. all by itself.
There is not a single real ceilling in sight, we only have clear barriers like compute but constant fast progress.
The field of mathematics went from 'useless' to 'you start better using it' to 'gamechanger' in how fast? 1 year after coding? less?
I want signes that we hit a real problem, instead I get cheaper tokens, Chinese models becoming very good as open models, new model updates from the others, mathematicans now saying how good it is etc.
Only half a year ago I had to babysit an LLM, now i tell it 1-3 sentences and it just goes and does it. And that stuff runs without compile errors etc.
If AI makes us 10% or 20% betteer, which is not that much, this alone will lead to companies reduing their expensive staff by 10-20%, which will has real impact on a job area. Some jobs are already hard to sell like cyber security and basic image tasks.
Hallucinations were low hanging fruit in some ways. As someone working on a large-ish complex-ish distributed system that has to be maintained and support customers, it's still very high value to have Claude in the mix, but the core problem of needing to monitor, advise, course correct, and make sure you don't end up with more code and complexity than you need is, at least in my experience, still roughly the same. The sharp edges are being filed off very rapidly, but the core experience of "make and maintain a large system" isn't advancing nearly as fast, IMO.
1 reply →
> The Hallucinations are becoming less, significantly by now.
Yes? What is the mega-solid technique that is used for it? Armies of people using curated data and reviewing it by hand? That is exactly one of my points: shifting the work elsewhere for specialized tasks. More replicable, improved, but, it scales infinitely and is autonomous? Can you assert that?
I am not denying there is some use (a lot of uses!) for this, but this is more nuanced than just: oh, they will replace us. Not at all, that day, with the current technology, is not going to arrive. This is just a systematization, fitting and tweaking of human knowledge by curated data. It is not the one true superintelligence they are selling us. To begin with, they do not have a concept of truth, but of probabilistic truth. Only that poses already a very, very big problem for the path to perfection.
> We have not seen the end of Reinforcement Learning, which does need a lot less training data but more compute.
Noone said the opposite, but I would like to know at which cost and if it is feasible. We do not have even enough compute power for current technology.
> . Its very capable of stringing together enough things so it can clearly follow handwavy things i tell it to do, analyse error messages, analysing screenshots etc. all by itself.
I use it every day for these tasks and it works well BECAUSE I review the output and makes me go faster. It finds a lot of things I would have not found and it also hallucinates another handful of them, which confirms my point about AIs not being able to be fully autonomous in any future point in time unless tweaked exactly for the task, and even then, it can still miss judgement a human could have for edge cases. So I am not sure of how bad or good it can be compared to a human but I am pretty sure it cannot be more reliable than an expert in many situations.
> Chinese models becoming very good as open models
I think they will be better in the long term if they follow this path. Not absolutely better but when mixing with economics and the fact that no frontier model is totally reliable anyway... why pay a lot for something that needs human inspection anyway?
> There is not a single real ceilling in sight, we only have clear barriers like compute but constant fast progress.
The ceiling is the paradigm itself, as I mentioned above. There is not a single chance with current technology that something could become "generically knowledgeable" and "reliable" both at the same time. If it becomes generically knowledgeable and reliable, it is bc of data fed into it and curated and tweaked by humans. This is not an original idea from myself, there are armies of people doing this every day around the world, you can check. This is where a lot of improvement comes from. Can this be reused? Of course. It is a generic solution? No way.
> Only half a year ago I had to babysit an LLM, now i tell it 1-3 sentences and it just goes and does it. And that stuff runs without compile errors etc.
Yes, I also do one-off scripts like this and code snippets, even reviews and others. Now go design a full distributed system. Use agents if you want. We come back in six months and compare it to a system that was properly written and tested by humans and we can compare the quality on some grounds:
You have to count all that money also, not just I vibe coded something and it seemed to work. With full systems things become super messy. Now add the human factor of requirements and back and forth (iterations can be admittedly faster with AI, especially prototypes, but that comes with other costs also)...
Not easy at all.
6 replies →
I think you meant to post this in response to https://news.ycombinator.com/item?id=49174900?
Yes I double checked the quote is not in the article. HN is probably the best place on the internet for people actually reading the article, but this being the top comment here suggests that the majority of voters still do not read the article
> I, as the human, still have to do the thinking as Claude still 'can't jump'
I still have to do quite a bit of thinking but the amount of of thinking I do per task is trending down. I agree LLMs are not good at abduction but very few humans are either and very few jobs/tasks require it. I can't talk for researchers jobs though. But perhaps fewer researchers would be desired by these labs (not none).
> I still have to do quite a bit of thinking but the amount of of thinking I do per task is trending down
Don't worry, I'm sure you'll hit your goal of zero thinking soon!
Trending towards the Homo Amens Mechanicus: the mechanical thoughtless human. What a goal
1 reply →
Cognitive offloading is delegating to the AI and still owning the answer. Cognitive surrender is when the AI’s output quietly becomes your output and there is nothing you feel is left to check. For software engineers the line between the two moves under your feet most days, and most of us are crossing it without noticing.
Source: https://addyosmani.com/blog/cognitive-surrender/
> I agree LLMs are not good at abduction but very few humans are either
I, too, am glad that few humans seem good at abduction.
And I hope LLMs don't get better at it.
"Abduction" being the way the article used it as in "explanatory reasoning in justifying hypotheses". https://plato.stanford.edu/entries/abduction/
Same reason some think preserving the environment is pointless because the believers will ascend to heaven, either way. It’s a religion. It’s dogmatic nihilism.
By trade I'm a UX Researcher/Designer who designs in code (HTML/CSS) and have done so since 2009. Recently I vibe coded an entire python app with a database and each time I didnt know what to do I would just feed screenshots to Gemini or Codex for guidance (i think i could share my screen with Codex and it can guide me via a voice conversation). I know I could follow up and build a companion iPhone and Android app using these tools.
Overall, I'd like to understand those who have a positive outlook on design and software engineering as a career. Where do you see the opportunity where I just see a bleak one where anyone can do this stuff by typing or talking to AI? Myself, after 17 years in the field I am begrudingly back in school for a new medical career. As well, anytime an IT recruiter reaches out I am getting responses back only after under-cutting the hourly rate I use to demand and what others probably are still trying to get. And with it feels even bleaker as it becomes a race to the bottom!
In my experience, not everyone can really do this stuff by typing. I think you need to be creative, resourceful, inventive, open minded and have ideas how to approach the typing/prompting. I see many people struggle in using AI.
> In my experience, not everyone can really do this stuff by typing. I think you need to be creative, resourceful, inventive, open minded and have ideas how to approach the typing/prompting. I see many people struggle in using AI.
The problem, for the profession, is that the set of people who can really do this stuff by typing is close to "all of them". I'm not seeing anyone struggle with using AI. I see struggles from professional software developers because they are trying to get quality output, but if you don't have a bar for quality, just about everyone can create their own software.
A poster a few months ago had a Show HN about his 7 year old kid, barely able to read, who was happily vibing up games.
1 reply →
The industry has vast (and increasing) oversupply of “programmers” versus diminishing demand. Add to this, the adoption of AI.
> Overall, I'd like to understand those who have a positive outlook on design and software engineering as a career.
I think until the market better achieves some equilibrium, there is no way general software programming (sorry “engineering”) should be considered as a career. That said, there will always be opportunities in particular markets or specialties.
I also work in UX and SWE, and heavily use GenAI in my work. I don’t have a positive outlook for people who limit their career to one of those fields, but I do have a positive outlook for generalist, multi-disciplinary careers. When you have the experience and skill to steer product development from end-to-end, you can produce high-quality products super-quickly. The experience and skills are the differentiator — if you lack those you can still use GenAI to move fast but probably in the wrong direction.
So one person now doing the job a handful use to do. That's what I hear you saying and Ive been thinking since my lay off in Feb. prompting me to be back in school.
It reminds me of Richard Hamming’s notorious question. I like the summary at https://bestjelly.substack.com/p/hamming-questions (which starts out with a quote from another site):
> > Mathematician Richard Hamming used to ask scientists in other fields "What are the most important problems in your field?" partly so he could troll them by asking "Why aren't you working on them?" and partly because getting asked this question is really useful for focusing people's attention on what matters.
> I imagine someone being asked this question, and how they should respond. I think like so - ‘Fuck off Richard’.
> This is partly because I imagine this question being asked in a kind of snarky, gotcha kind of way, with some sort of nerdy superiority. Like ‘ha your behaviour is inconsistent with your implied preferences, you idiot, do you even von Neumann–Morgenstern?’
https://gwern.net/doc/science/1986-hamming#the-importance-of... https://theonion.com/study-average-person-s-life-plan-can-on...
although, if i'm out of tokens and have to wait a full day, i won't bother doing some things manually because the day i'll spend doing something won't take more than 1 hour the next day when tokens are available again.
That seems like a somewhat orthogonal point? Like, if I'm a carpenter and my batteries all run out / I can't actually power my power tools then the best course of action is to go home and recharge all the batteries instead of trying to hand-cut 100 pieces of lumber today. After all, the power tools can do it a lot faster (and with less effort) than I can.
I say this as someone who's watched a bunch of woodworking videos but hasn't actually done this myself :)
I read that more so as, I'm a carpenter and my batteries have all ran flat, so I'll put them on charge and do something else today. I'll cut up the lumber tomorrow when the batteries have charged.
According to the article that part being automated isn’t more than 18% of your day anyway
At this point at least half of your day is spent doing work that you shouldn't be doing in the first place, but over past decades companies saw it fit to eliminate specialized roles with legible paychecks, and smear the work they did on everyone else until it disappears from the books.
Self-service and office suite software is largely responsible for this.
Also people tend to forget that LLMs still just work on compressed data... Where are the MAJOR breakthroughs? Where is all the "crazy" AI output going? Software seemed to degrade in quality a lot in the recent years. All "improvements" LLMs go through are simply improvements on how to burn more tokens out of my pockets given that Claude now want an actual browser extension to "visually" confirm small changes every time I use it for UI. They are still just data parrots.
From what I see most benefits are for people that work with LLMs, but usually smaller percentages never 50% or more because of the LLMs (OK, unless you were doing basic, repetitive stuff, but then that's not to write about).
Which kind of answers the original question "why bother working?" with "because now, I can do a bit more than before".
I also see bad quality (in code, documents, presentations). It comes from people that had no clue how to do something before and now they imagine that just asking Claude is solving well the problem. And is annoying (and hard) to explain to it them, and then they get frustrated.
LLM don't work on 'compressed data'. LLM compress data into their latent space which allows them to become general.
They learn the concept of things and how to do them because this is better compression than learning concepts one by one.
Which means, if an LLM 'learns' the concept of a poem, it can put everything into the formad of a poem instead of learning a billion poems.
> They learn the concept of things and how to do them because this is better compression than learning concepts one by one.
When anthropic looked at how an LLM does addition it found it had some mental math heuristics that might or might not always work. The LLM hadn't learned the concept of addition. It had learned some heuristics that might work for some numbers. The result is that LLM's cannot add numbers reliably because they have not learned the concept of addition.
1 reply →
There’s a small group of established architects talking about harness engineering, but I’m not sure anyone is actually listening to them.
And those same architects are quietly extracting real productivity from GenAI.
And even this write up skips that info by waving, “Some people…”
Mind sharing some names or something else we can learn about?
2 replies →
Idk, maybe some people get crazy productivity out of LLMs. To me, going deep into the AI bubble, reading about terms I've never seen before just feels like some crypto bro bubble with people being too deep into the sauce to notice that these things are not the wonder machines they believe so hard in...
The only thing I noticed in some of the software I use is more design changes, but they are often not better than before.
> And I say that as someone who uses Claude Code in complex environments almost hourly; I, as the human, still have to do the thinking as Claude still 'can't jump' [1] and I have seen no evidence that they (or similar AI, any time soon) will 'jump' like a human brain does.
Sure it can, turn up the "temperature" a bit.
There's this notion that human "jumping" is magic. It's not. It's all based on inputs. Including unrelated inputs, past inputs, and feeding yourself your own thoughts.
The hard part is not the ability to make conceptual jumps. That's just random search. The hard part is discrimination: whether a given mental jump is "creative" or "insane". Iterated, the problem is that of balancing between the two failure modes: relax your thinking too much, and you'll start thinking nonsense thoughts; tighten it too much, and you'll be just following immediate-term rewards and obvious thought trains. It takes time to find that balance, and plenty of people at various points err in one or the other direction (e.g. small kids in particular tend to err on the "crazy non-sequitur side", but that's because they're learning the basics of reality and social interactions).
100% agree. Plus you can easily instruct it to use inspiration from unspecified unrelated and counterintuitive concepts at random and in parallel. The "jump" is not "on" by default because it would waste tokens on high risk paths, not because it's incapable. It's very capable if you are willing to wast some tokens.
> The hard part is discrimination: whether a given mental jump is "creative" or "insane"
That's what they mean by LLM's can't jump. They mean it can't make a creative jump. Their example is Einstein's Theory of Relativity - It's not a random jump.
Einstein didn't magically one day woke up and jump on relativity theory. He had prerequisites in terms of recent mathematical advancements (notation) and physical discoveries, and a job exposing him to a lot of lateral thinking, and time to bounce ideas around in his head. We don't know how many fruitless jumps he made before making one that we remember him for.
1 reply →
A fully automated utopia isn't just going to happen. Even with frontier models, the integrations, the evals, the UX, need a lot of work and someone needs to do it. After I've automated this thing I'll move on to the next task, this is what it means to be a software engineer.
An actual utopia would require never-before-seen democratic mandate from people who are currently on the brink of hot civil war
It still makes a massive difference for me if they only need a handfull people now.
Generating a good looking UI for example, is so much easier now with LLM.
For a joke I asked ChatGPT yesterday to make a short promoimage for a 'joke' idea i had, it was above avg. I have for sure seen worse Marketing Images than what ChatGPT generated.
It looked similiar to plenty of other Marketing Images but its not that anyone cares.
Robot powered factories still have humans there, the gist is that they are a tiny fraction of what a classical factory would require 50 years ago.
AI has made 'jumps' in demanding fields like leading mathematical research and has made advancements in AI research itself. Is now a good time to start a maths career? Is there a field of research (yours?) which is inherently (more) AI proof?
Btw, I think the discussion of Einstein's career in the paper you link is historically wrong in many respects, particularly the argument about 'weak signal'. Einstein was in fact working on some of the most mainstream and widely discussed problems in physics of the day, he is admired for the creativity of his solutions to those problems, and much of his work built incrementally on ideas and breakthroughs that came (long) before (as all research does).
Article suggests that a central motivation of Einstein's work was resolving action-at-a-distance in Newtonian mechanics - yet Maxwell introduced the same Lagrangian field theories for electromagnetism we use today 50 years earlier to solve the same problem for Farraday's laws of electromagnetism. Similar wave equations existed even earlier. Heaviside in 1893 extended this technique to gravity (matching 'weak field' GR) 20 years earlier. So this is perhaps the one aspect of gravity that had actually already been solved before Einstein. Authors might be conflating his work on action-at-a-distance in QM.
Einstein's GR extended the linear 'weak field' understanding of gravity to include the non-linear self-referential case where masses themselves create gravity. This was mathematically incredibly difficult but was necessary precisely because SR's mass energy equivalence created so many strong signals that were unresolved. For example: if finite energy is mass, then mass changes as objects accelerate past a large mass like a start, and hence their propagation in space could not be explained by linear EM style field equations. Many such considerations were causing very 'strong signals' in SR, and there were analogous problems in QM atomic models being developed at the same time.
SR was also a solution to a problem that was actively being worked by many of the leading physicists of the day. SR actually does match Newtonian mechanics for a single observer - it resolves contradictions in the case of separate observers, by allowing them to assign different values to the speeds, masses, etc of objects such that each object appears to follow Newtonian mechanics for each observer. Again, this was necessary because of a lot of contradictions related to the behavior of light that had been well-known for ~20 years at the time.
Personally, I don't consider this kind of reasoning to be beyond the capabilities of future LLMs (even current LLMs if the task was broken into technical rather than philosophical problems). Personally, I doubt that such problems could stand open for 20+ years waiting for a creative genius to solve them in the modern world.
And don't get me started on the philosophy.
Also, if you believe your well-paying job is eventually going to be automated you would be prudent to bank the money while you prepare for the future.
I completely agree. There's no guarantee that researchers won't still be needed despite AI.
Sorry where in the article does it make this point? I cannot find the text "On my visits to the Bay Area" anywhere in the original article.
It doesn't, GP was responding to the Gwern story at https://news.ycombinator.com/item?id=49174900 but posted in the wrong place.
It also seems kinda tone deaf. If someone basically told me I was wasting my time and asked what I would do in the future, I would not bother giving them a particularly thoughtful answer because trying to spend effort justifying my life choices to them would be the actual waste of time.
What kind of answers were they expecting to get?