Comment by Verdex
6 days ago
Hundreds of comments. Some say LLMs are the future. Others say they don't work today and they won't work tomorrow.
Videogame speed running has this problem solved. Livestream your 10x engineer LLM usage, a git commit annotated with it's prompt per change. Then everyone will see the result.
This doesn't seem like an area of debate. No complicated diagrams required. Just run the experiment and show the result.
I literally do this about twice a week on Twitch. I’m working on some hobby projects, and one constraint I’ve put on myself is to use LLMs for pretty much everything, regardless of whether it saves me time or not. The reason is twofold: I want to learn how to use them efficiently, and I want to constantly test the waters to see where the limits of their capabilities are. On my stream, you can see LLMs fail spectacularly one time and do hours of work in minutes another time.
I’m not alone in this - there are tons of other examples of people showing how they use LLMs online; you just need to search for them.
Thanks. I'll give them a watch.
Agreed.
The article provides zero measurement, zero examples, zero numbers.
It's pure conjecture with no data or experiment to back it up. Unfortunately conjecture rises to the top on hackernews. A well built study on LLM effectiveness would fall off the front page quickly.
I'd honestly love to see this.
People always say "you just need to learn to prompt better" without providing any context as to what "better" looks like. (And, presumes that my prompt isn't good enough, which maybe it is maybe it isn't.)
The easy way out of that is "well every scenario is different" - great, show me a bunch of scenarios on a speed run video across many problems, so I can learn by watching.
It's because you get to the No True Scotsman -thing pretty fast.
If I use LLMs to code, say a Telegram bot that summarise the family calendars and current weather to a channel - someone will come in saying "but LLMs are shit because they can't handle this very esoteric hardware assembler I use EVERY DAY!!1"
But... do you know anybody who will give me 50k a year to write telegram bots for them?
1 reply →
I'd like to think I could dismiss those responses and still learn something from a collection of samples.
The Primeagen did this. He locked himself in for 7 weeks and tried to build a game by only using cursor. The videos are more entertainment style but this is the closest thing i know of
https://www.youtube.com/watch?v=6x3weFV9YI4
AI Coding is becoming an edge, and sharing your edge isn't the wisest thing to do, even more so when doubt is so prevalent!
Extremely doubtful.
This thread has hundreds of comments where people are screaming that everyone needs to learn AI coding.
If it was such an edge would they not otherwise keep quiet?
Because there are forces that are trying to kill the momentum.
Imagine that there was a serum that gives you superhuman strength only under specific conditions that you’re supposed to discover. Then there’s half room who screams that it should be banned, because it is cheating/fake/doesn’t work. And there’s another half room that swears by it, because they know how to utilize it properly.
You know it works and you don’t want to give up your secret sauce or make another half of the room stronger.
5 replies →
Or they just aren't sociopaths, so they don't think that way?
Quiet doesn't bring your AI corporation profits up.
Unlikely. Programming in highly collaborative and efficiency is hard to measure. That creates incentives for programmers in competition to typically prioritize advertising their skills by demonstration over maintaining their secret edge. Be it at work or on the internet, if you help others by sharing your techniques you'll make them want to work with you and impress them with how smart you are. If you are keeping it all secret to maintain your edge, people will think of you as unhelpful and they won't know how smart you are, because it's very difficult to judge how difficult the things our accomplished were. The reason people don't stream themselves vibe coding is that's it's even less interesting to watch than regular coding.
It's not your edge anymore. It's AI's. And how is TFA any different?
Someone will always be idealistic enough to share. The fact that we do not see them now should raise a few eyebrows.
I must disagree. Sharing your edge is the wisest possible thing you can do on a societal level. For a slightly silly idea would it be better to have say, everyone doing guesswork knots for how to tie their shoes vs a single reliable 'rabbit ears' technique? Then you can see the benefits to having edges widely shared as a norm. That is the foundation of how society can learn.
Just like blockchain, smart contracts and web 3.0 are the future!
I see all the negative responses, but this seems true to me. I am old enough to remember the dot com days and could see the transformative effect of the Internet from miles away when I was a teenager. Yet many, many people refused to acknowledge that someday soon we would do things like credit card transactions online, or that people might buy shoes without trying them on first, etc.
You could say it is a lack of imagination or not connecting the dots, but I think there is a more human reason. A lot of people don't want the disruption and are happy with the status quo. I'm a software engineer so I know how problematic AI may be for my job, but I think anyone who looks at our current state and the recent improvements should be able to see the writing on the wall here.
I for one am more curious than afraid of AI, because I have always felt that writing code was the worst part of being a programmer. I am much happier building product or solving interesting problems than tracking down elusive bugs or refactoring old codebases.
I disagree with that. I was around when the web grew into the mainstream, and almost everybody was sure that it would have a huge impact on every industry and activity it touched. There wasn't even remotely a level of skepticism comparable to those around VR, blockchain, and now GenAI.
And it seems pretty obvious why. The benefits were clear and palpable. Communication was going to become a heck of a lot easier, faster, cheaper, barriers were being lowered.
There's no such qualitative advantage offered by GenAI, compared to the way we did things before. Web vs. pre-Web, the benefits were clear.
GenAI? Some execs claim it's making stuff cheaper, but it doesn't consider quality and long-term effects, plus it's spouted by those with no technological knowledge and with a reputation to long have cashed out and moved on by the time their actions crash a company. Plus, still nobody seems to have figured out how to make money (real money, not VC) off of this. Faster -- again, at what price to quality?
Then there's the predictions. We've been told for about three years now about the explosive rise in quality we'll see from GenAI output. I'm still waiting. The predictions of wider spread, higher speed and lower cost of the web sounded plausible, and they materialised. Comparatively, I see a lot of very well-reasoned arguments for the hypothesis that GenAI has peaked (for now) and this is pretty much as good as it's going to get, with source data sets exhausted and increasingly polluted by poor GenAI slop. So far, the trajectory makes me believe this scenario to be a lot more likely.
None of this seems remotely comparable to the Internet or web cases to me. The web certainly didn't feel like a hype to me in the 90s and I don't remember anyone having had that view.
So, programmers once had an edge in having their source closed, then fell for the open source evangelism and started sharing their code, which enabled the training of AI models, and now the next iteration of what was called programmers before and is now known as vibe coders has this notion of having an edge in having their chatbot prompts closed again?
Let's all just muse some and imagine what the next cycle of this wheel will look like.