Comment by stego-tech
7 days ago
On the one hand, I would expect LLMs to be able to crank out such code when prompted by skilled engineers who also understand prompting these tools correctly. OAuth isn’t new, has tons of working examples to steal as training data from public projects, and in a variety of existing languages to suit most use cases or needs.
On the other hand, where I remain a skeptic is this constant banging-on that somehow this will translate into entirely new things - research, materials science, economies, inventions, etc - because that requires learning “in real time” from information sources you’re literally generating in that moment, not decades of Stack Overflow responses without context. That has been bandied about for years, with no evidence to show for it beyond specifically cherry-picked examples, often from highly-controlled environments.
I never doubted that, with competent engineers, these tools could be used to generate “new” code from past datasets. What I continue to doubt is the utility of these tools given their immense costs, both environmentally and socially.
It's said that much of research is data janitorial work, and from my experience that's not just limited to the machine learning space. Every research scientist wishes that they had an army of engineers to build bespoke tooling for their niche, so they could get back to trying ideas at the speed of thought rather than needing to spend a day writing utility functions for those tools and poring over tables to spot anomalies. Giving every researcher a priceless level of leverage is a tremendous social good.
Of course, we won't be able to tell the real effects, now, because every longitudinal study of researchers will now be corrupted by the ongoing evisceration of academic research in the current environment. Vibe-coding won't be a net creativity gain to a researcher affected by vibe-immigration-policy, vibe-grant-availability, and vibe-firings, for all of which the unpredictability is a punitive design goal.
Whether fear of LLMs taking jobs has contributed to a larger culture of fear and tribalism that has emboldened anti-intellectual movements worldwide, and what the attributable net effect on research and development will be... it's incredibly hard to quantify.
> Vibe-coding won't be a net creativity gain to a researcher affected by vibe-immigration-policy, vibe-grant-availability, and vibe-firings, for all of which the unpredictability is a punitive design goal.
Quite literally this is what I’m trying to get at with my resistance to LLM adoption in the current environment. We’re not using it to do hard work, we’re throwing it everywhere in an intentional decision to dumb down more people and funnel resources and control into fewer hands.
Current AI isn’t democratizing anything, it’s just a shinier marketing ploy to get people to abandon skilled professions and leave the bulk of the populace only suitable for McJobs. The benefits of its use are seen by vanishingly few, while its harms felt by distressingly many.
At present, it is a tool designed to improve existing neoliberal policies and wealth pumps by reducing the demand for skilled labor without properly compensating those affected by its use, nor allowing an exit from their walled gardens (because that is literally what all these XaaS AI firms are - walled gardens of pattern matchers masquerading as intelligence).
This is a bit stronger than my point, I should say. I do think that LLMs would have a net benefit to society, by way of their effects on research and innovation... if we could get our political houses in order such that we weren’t negating those effects, and such that we were empowering small businesses and high-tech startups to build with the results of this innovation sustainably.
And in a world where policy is horrid and the effects are mainly negated, things would be even worse if the remaining researchers lost AI as a tool. For better or for worse, fire has been shared with humanity, and we might as well cook.
>Current AI isn’t democratizing anything
I don't work anywhere close with software but I have used chatgpt to program small tools and scripts for me I never would have written myself.
The real boon of AI programming is when normal people use it to program things custom tailored for their use case.
That’s one perspective, but it’s wrong and typical gatekeeping (do you have a software degree by any chance?). People had the same attitude towards open source tooling and low code frameworks - god forbid someone not certified and ordained build a solution in something other than Java...
AI code tools are allowing people to build things they couldn't before due to lack of skillset, time or budget. I’ve seen all sorts of problems solved by semi technical and even non-technical people. My brother for example built a thing with Microsoft copilot that helped automate more in his manufacturing facility (used to be paper).
But yeah, keep yelling at that cloud - the rest of us will keep shipping cool things that we couldn’t before, and faster.
11 replies →
This is one of the best comments about the current AI hype.
The elite really don't see why the proletariat should be interested in, or enjoy the dignity of, actual skill and quality.
Hence the enshitification of everything, and now AI promises to commoditize everything into slop.
Sad because it is the very deoth of society that has birthe
7 replies →
There's too much "close enough" in virtually all these discussions. LLM is not a hand grenade. It's important to keep in mind what LLMs and related tech can be relied upon do or assist with, can't be relied upon to do or assist with, and might be relied on to do in the future.
> rather than needing to spend a day writing utility functions for those tools and poring over tables to spot anomalies
A ridiculous amount of most researchers' time is spent cleaning up data.
> much of research is data janitorial work
In applied research perhaps, Fundamental research is nothing like that in any field including ML.
> Fundamental research is nothing like that in any field including ML
Grant writing.
All experimental or empirical research is like that, is closer to the point.
I like to make a rough analogy with autonomous vehicles. There's a leveling system from 1 (old school cruise control) to 5 (full automation):
* We achieved Level 2 autonomy first, which requires you to fully supervise and retain control of the vehicle and expect mistakes at any moment. So kind of neat but also can get you in big trouble if you don't supervise properly. Some people like it, some people don't see it as a net gain given the oversight required.
^ This is where Tesla "FSD beta" is at, and probably where LLM codegen tools are at today.
* After many years we have achieved a degree of Level 4 autonomy on well-trained routes albeit with occasional human intervention. This is where Waymo is at in certain cities. Level 4 means autonomy within specific but broad circumstances like a given area and weather conditions. While it is still somewhat early days it looks like we can generally trust these to operate safely and ask for help when they are not confident. Humans are not out of the loop.[1]
^ This is probably what where we can expect codegen to grow after many more years of training and refinement in specific domains. I.e. a lot of what CloudFlare engineers did with their prompt engineering tweaking was of this nature. Think of them as the employees driving the training vehicles around San Francisco for the past decade. And similarly, "L4 codegen" needs to prioritize code safety which in part means ensuring humans can understand situations and step in to guide and debug when the tool gets stuck.
* We are still nowhere close to Level 5 "drive anywhere and under any conditions a human can." And IMHO it's not clear we ever will based purely on the technology and methods that got us to L4. There are other brain mechanisms at work that need to be modeled.
[1] https://www.cnbc.com/2023/11/06/cruise-confirms-robotaxis-re...
That's a good analogy. OAuth libraries and integrations are like a highly-mapped California city. Just because you can drive a Waymo or coding agent there, doesn't mean you can drive it through the Rockies.
Note that even with OAuth it took, as of today, security engineers many iterations of review and prompt tweaking to get this result. We’re still in the “mapping” phase.
> On the other hand, where I remain a skeptic is this constant banging-on that somehow this will translate into entirely new things - research, materials science, economies, inventions, etc
Does it even have to be able to do so? Just the ability to speed up exploration and validation based on what a human tells it to do is already enormously useful, depending on how much you can speed up those things, and how accurate it can be.
Too slow or too inaccurate and it'll have a strong slowdown factor. But once some threshold been reached, where it makes either of those things faster, I'd probably consider the whole thing "overall useful". Nut of course that isn't the full picture and ignoring all the tradeoffs is kind of cheating, there are more things to consider too as you mention.
I'm guessing we aren't quite over the threshold because it is still very young all things considered, although the ecosystem is already pretty big. I feel like generally things tend to grow beyond their usefulness initially, and we're at that stage right now, and people are shooting it all kind of directions to see what works or not.
> Just the ability to speed up exploration and validation based on what a human tells it to do is already enormously useful, depending on how much you can speed up those things, and how accurate it can be.
The big question is: is it useful enough to justify the cost when the VC subsidies go away?
My phone recently offered me Gemini "now for free" and I thought "free for now, you mean. I better not get used to that. They should be required to call it a free trial."
Inference is actually quite cheap. Like, a highly competitive LLM can cost 1/25th of a search query. And it is not due to inference being subsidized by VC money.
It's also getting cheaper all the time. Something like 1000x cheaper in the last two years at the same quality level, and there's not yet any sign of a plateau.
So it'd be quite surprising if the only long-term business model turned out to be subscriptions.
7 replies →
> The big question is: is it useful enough to justify the cost when the VC subsidies go away?
I won't claim local LLMs as nearly as good as various top models behind paid subscriptions/APIs, but I'm certain I'd be able find a way (for me) of working with them well enough, if the entire paid/hosted ecosystem disappeared over night. Even with models released today.
I think the VC subsidies probably "make stuff happen" faster, and without it we'd see slower progress, but I don't think 100% of the ecosystem would disappear even if 100% of VC funding disappeared. We're bound for another AI winter at one point, and some will surely survive even that :)
So isn't the heuristic that if your job is easily digestible by an LLM, you're probably replaceable, but if the strong slowdown factor presents itself, you're probably doing novel work and have job security?
> So isn't the heuristic that if your job is easily digestible by an LLM, you're probably replaceable
Yeah, that sounds about right to me. I wasn't talking about wholesale replacement though, but as a tool/augmentation, I'm not very confident an LLM would be able replace a software engineer, but I can definitely see many workflows of a software engineer being sped up, like the exploration and validation process.
> On the other hand, where I remain a skeptic is this constant banging-on that somehow this will translate into entirely new things
Really a lot of innovation, even at the very cutting edge, is about combining old things in new ways, and these are great productivity tools for this.
I've been "vibe coding" quite a bit recently, and it's been going great. I still end up reading all the code and fixing issues by hand occasionally, but it does remove a lot of the grunt work of looking up simple things and typing out obvious code.
It helps me spend more time designing and thinking about how things should work.
It's easily a 2-3x productivity boost versus the old fashioned way of doing things, possibly more when you take into account that I also end up implementing extra bells and whistles that I would otherwise have been too lazy to add, but that come almost for free with LLMs.
I don't think the stereotype of vibe coding, that is of coding without understanding what's going on, actually works though. I've seen the tools get stuck on issues they don't seem to be able to understand fully too often to believe that.
I'm not worried at all that LLMs are going to take software engineering jobs soon. They're really just making engineers more powerful, maybe like going from low level languages to high level compiled ones. I don't think anyone was worried about the efficiency gains from that destroying jobs either.
There's still a lot of domain knowledge that goes into using LLMs for coding effectively. I have some stories on this too but that'll be for another day...
> where I remain a skeptic is this constant banging-on that somehow this will translate into entirely new things - research, materials science, economies, inventions, etc - because that requires learning “in real time” from information sources you’re literally generating in that moment, not decades of Stack Overflow responses without context.
Personally I hope this will materialize, at the very least because there's plenty of discoveries to be made by cross-correlating discoveries already made; the necessary information should be there, but reasoning capability (both that of the model and that added by orchestration) seems to be lacking. I'm not sure if pure chat is the best way to access it, either. We need better, more hands-on tools to explore the latent spaces of LLMs.
I don’t consider that “new” research, personally - because AI boosters don’t consider that “new”. The future they hype is one where these LLMs can magic up entirely new fields of research and study without human input, which isn’t how these models are trained in the first place.
That said, yes, it could be highly beneficial for identifying patterns in existing research that allows for new discoveries - provided we don’t trust it blindly and actually validate it with science. Though I question its value to society in burning up fossil fuels, polluting the atmosphere, and draining freshwater supplies compared to doing the same work with Grad Students and Scientists with the associated societal feedback involved in said employment activities.
> Though I question its value to society in burning up fossil fuels, polluting the atmosphere, and draining freshwater supplies compared to doing the same work with Grad Students and Scientists with the associated societal feedback involved in said employment activities.
I'd imagine AI is much cheaper on that front than grad students, whether you count marginal contribution, or total costs of building and utilization. Humans are damn expensive and environmentally intensive to rear and keep around.
2 replies →
most engineering is glorified plumbing so as far as labour productivity goes, this should go a long way
I doubt it, for the simple reason that literal plumbers still make excellent money because plumbing is ultimately bespoke output built on standards.
Everyone wants to automate the (proverbial) plumbing, until shit spews everywhere and there’s nobody to blame but yourself.
Plumbers make excellent money because regulations require licensed plumbers to do the work, and plumbing unions have a financial interest in limiting the number of plumbers.
But anybody can do plumbing. It’s not rocket science.
10 replies →
I have a non-zero number of industrial process patents under my belt. Allegedly, that means that I had ideas that had not previously been recorded. Once I wrote them down, paid some lawyers a bunch of money, and did some paperwork, I have the right to pay lawyers more money to make someone's life difficult if I think that someone ever tries to do something with the same thoughts, regardless of if they had those thoughts before, after, or independently of me.
In my opinion, there is a very valid argument that the vast majority of things that are patented are not "new" things, because everything builds on something else that came before it.
The things that are seen as "new" are not infrequently something where someone in field A sees something in field B, ponders it for a minute, and goes "hey, if we take that idea from field B, twist it clockwise a bit, and bolt it onto the other thing we already use, it would make our lives easier over in this nasty corner of field A." Congratulations! "New" idea, and the patent lawyers and finance wonks rejoice.
LLMs may not be able to truly "invent" "new" things, depending on where you place those particular goalposts.
However, even a year or two ago - well before Deep Research et al - they could be shockingly useful for drawing connections between disparate fields and applications. I was working through a "try to sort out the design space of a chemical process" type exercise, and decided to ask whichever GPT was available and free at the time about analogous applications and processes in various industries.
After a bit of prodding it made some suggestions that I definitely could have come up on my own if I had the requisite domain knowledge, but would almost certainly never have managed on my own. It also caused me to make a connection between a few things that I don't think I would have stumbled upon otherwise.
I checked with my chemist friends, and they said the resulting ideas were worth testing. After much iteration, one of the suggested compounds/approaches ended up generating the least bad result from that set of experiments.
I've previously sketched out a framework for using these tools (combined with other similar machine learning/AI/simulation tools) to massively improve the energy consumption of industrial chemical processes. It seems to me that that type of application is one where the LLM's environmental cost could be very much offset by the advances it provides.
The social cost is a completely different question though, and I think a very valid one. I also don't think our economic system is structured in such a way that the social costs will ever be mitigated.
Where am I going with this? I'm not sure.
Is there a "ghost in the machine"? I wouldn't place a bet on yes, at least not today. But I think that there is a fair bit of something there. Utility, if nothing else. They seem like a force multiplier to me, and I think that with proper guidance, that force multiplier could be applied to basic research, material science, economics, and "inventions".
Right now, it does seem that it takes someone with a lot of knowledge about the specific area, process, or task to get really good results out of LLMs.
Will that always be true? I don't know. I think there's at least one piece of the puzzle we don't have sorted out yet, and that the utility of the existing models/architectures will ride the s-curve up a bit longer but ultimately flatten out.
I'm also wrong a LOT, so I wouldn't bet a shiny nickel on that.