Comment by subdavis
4 days ago
I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet.
I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders and worked on the first try.
Afterward, I sort of had to reflect on the fact that I learned essentially nothing about building vector search. I wanted the feature more than I wanted to know how to build the feature. It kept me learning the thing I cared about rather than doing a side quest.
I don't think building it the long way is necessarily a more effective way to learn.
You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works.
My hunch is that the 30 minutes of focused learning spent with a custom-built version that solves your exact problem is as effective (or even more effective) than four hours spent mostly struggling to get something up and running and going down various rabbit holes of unrelated problem-solving.
Especially if realistically you were never going to carve out those four hours anyway.
This feels like the exactly wrong way to think about it IMO. For me “knowledge” is not the explicit recitation of the correct solution, it’s all the implicit working knowledge I gain from trying different things, having initial assumptions fail, seeing what was off, dealing with deployment headaches, etc. As I work, I carefully pay attention to the outputs of all tools and try to mentally document what paths I didn’t take. That makes dealing with bugs and issues later on a lot easier, but it also expands my awareness of the domain, and checks my hubris on thinking I know something, and makes it possible to reason about the system when doing things later on.
Of course, this kind of interactive deep engagement with a topic is fast becoming obsolete. But the essence to me of “knowing” is about doing and experiencing things, updating my bayesian priors dialectically (to put it fancily)
I agree that the only reliable way to learn is to put knowledge into practice.
I don't think that's incompatible with getting help from LLMs. I find that LLMs let me try so much more stuff, and at such a faster rate, that my learning pace has accelerated in a material way.
8 replies →
I remember a very nice quote from an Amazon exec - “there is no compression algorithm for experience”. The LLM might as well do wrong things, and you still won’t know what you don’t know. But then, iterating with LLMs is a different kind of experience; and in the future people will likely do that more than just grinding through the failure of just missing semicolons Simon is describing below. It’s a different paradigm really
4 replies →
But how much of that time is truly spent on learning relevant knowledge, and how much of it is just (now) useless errata? Take vector search for an example. Pre-GPT, I would spend like an hour chasing down a typo, like specifying 1023 instead of 1024 or something. This sort of problem is now trivially solved in minutes by a LLM that fully understands the API surface area. So what exactly do I lose by not spending that hour chasing it down? It has nothing to do with learning vector search better, and an LLM can do it better and faster than I can.
4 replies →
I think is exactly right in principle and practically. The question is what domain knowledge you should improve on to maximize outcome: Will understanding the machine code be the thing that most likely translates to better outcomes? Will building the vector search the hard way be? Or will it be focusing on the thing that you do with the vector search?
At some point things will get hard, as long as the world is. You don't need to concern yourself with any technical layer for that to be true. The less we have to concern ourselves with technicalities, the further that points shifts towards the thing we actually care about.
Forgetting LLMs and coding agents for a second, what OP describes is like watching a Youtube video on how to make a small repair around the house. You can watch that and "know" what needs to be done afterwards. But it is a very different thing to do it yourself.
Ultimately it comes to whether gaining the know how through experience is worth it or not.
1 reply →
Trial and error is not how apprenticeship works, for example.
As an apprentice, you get correct and precise enough instructions and you learn from the masters perfection point downwards.
Maybe we have reached a point where we can be the machine's apprentices in some ways.
You could say that knowledge is understanding all the paths that won't solve your problem.
Take a look at Bloom's taxonomy. It's exactly about what you are talking about.
Just speaking from personal experience but the struggle is what creates the learning.
I learned refactoring patterns from Fowler's book. But when I tried to actually use them I still struggled. I didn't fully understand how the patterns worked until I actually tried (and failed) to use them a few times.
You don't really internalize things until you understand what doesn't work just as much as what does. You don't learn nearly as much from success as you do from failure. I would say the ratio of truly internalized knowledge is much higher for failure.
The notion that you can get a bot to just vomit out a vector database and then you can just "read the code" and you'll understand how a vector database works is just ludicrous.
This conversation isn't about building a vector database from scratch, it's about learning to integrate with an existing vector database.
5 replies →
> Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works
I can only speak for myself, but the only way I've been able to learn things rapidly in this industry is by writing things myself: even rote re-typing of books or SO answers was enough to trigger this for me.
Just querying models and reading output doesn't seem to work for me, but that's maybe down to my particular learning style.
That's why I said "tweaking it yourself" - that's the point where you go beyond "just querying models and reading output".
1 reply →
>>My hunch is that the 30 minutes of focused learning spent with a custom-built version that solves your exact problem is as effective
My hunch is the exact opposite of this. You will learn close to nothing by reading this for 30 minutes.
That's assuming everyone learns the same way, which isn't true. Watching a streamer beat a dark souls boss won't automatically make you competent at the game. Reading through gobs of code generated for you without knowing why various things were needed won't help either. A middle approach could be to get the LLM to guide you through the steps.
It's not just assuming that everyone learns the same way. It's assuming that everyone learns the way that all of the research literature on learning claims does not work.
Learning requires active recall/synthesis. Looking at solved examples instead of working them yourself does not suffice in math, physics, chemistry, or CS, but somehow it is supposed to work in this situation?
I don't know. I built a vector similarity system for my hobby project the "hard" way, which was mostly getting Python set up with all the dependencies (seriously, Python dependency resolution is a non-trivial problem), picking a model with the right tradeoffs, installing pgvector, picking an index that optimized my distance metric, calculating and storing vectors for all my data, and integrating routes and UI which dispatched ANN search (order by / limit) to my indexed column. I also did some clustering, and learned something of how awkward it is in practice to pick a representative vector for a cluster - and in fact you may want several.
I now know what the model does (at a black box level) and how all the parts fit together. And I have plans to build classifiers on top of the vectors I built for further processing.
The experience of fighting Python dependencies gives me more appreciation for uv over venv and will leave me less stuck whenever the LLM fails to help resolve the situation.
It's the same hunch we all have when we think we're going to learn something by watching tutorials. We learn by struggling.
I buy the productivity argument, but I’m not convinced “30 minutes reading/tweaking agent output” is equivalent for learning to building it yourself.
If your goal is the feature, then yes: letting the agent do the heavy lifting and reviewing the diff afterward is a huge win.
But if your goal is understanding / skill-building, the hard part usually isn’t seeing a working solution. It’s doing the messy work of (a) making design choices, (b) getting stuck, (c) debugging, and (d) forming the mental model that lets you reproduce it later. Reviewing a correct implementation can create a feeling of “I get it,” but that feeling often doesn’t survive a blank file.
I’ve noticed this in my own hobby coding: LLMs are great for familiarity and unblocking progress, but the learning “sticks” much more when I’ve had to struggle through the failure modes myself. I’m watching the same dynamic play out with my son using ChatGPT to study for physics/calculus . . . it feels deep for him in the moment with the LLM, but exam-style transfer exposes the gaps.
If I had four hours to dedicate to this particular learning project I would still use LLMs to help me along the way, with the expectation that I'd learn more from those four hours than I would if I'd spent the same amount of time deliberately not using LLMs to help me.
We've been given a tool that lets us ask questions in human language and get back answers that are correct 90% of the time! And that remaining 10% means we have to engage critically with those answers, which is a useful learning trick in its own right.
2 replies →
Reading without actually doing does not really result in learning, only very marginal one.
Try reading tutorials on a new programming language for 30 minutes and then open new text file and write basic loop with print.
It won’t even compile- which shows you haven’t really learned anything. Just read an interesting story. Sure you pita few bits here and there but you still don’t know how to do even the moat basic thing.
Working with an LLM feels very different to me from reading a static tutorial.
It's more like having the tutorial author there with you and actively engaging with them to collaborate on building the exact tutorial for the exact project you're looking to build.
I'll take a bidirectional conversation with a subject matter expert (and when you're just staring to learn Rust LLMs can absolutely take the role of "expert", in comparison to you at least) over struggling on my own against static documentation and the Rust compiler.
And I can take over the wheel at any moment! It's entirely on me to decide how much I get to do vs how much the LLM does for me.
That's why learning in this way is a skill in its own right, and one that I'd like to see studied and formalized and taught to other people.
This really makes for a good natural experiment: carry on :)
I have a hard time imagining how much you'd have to literally bribe me to get me to try doing it the way you describe. I'm too interested in implementation details of things and looking for innovations—in fact I make my living doing that, like some cyberpunk gremlin just delighting in messing with stuff in unexpected ways. I don't understand why you're not, but maybe it's not for me to understand.
Carry on. We'll check back and see how it worked for ya :)
I delight in messing with stuff in unexpected ways, and see that as a great way to learn. I just don't want to have to type all of that mischief out by hand.
I ported a complex new Python program I wrote to Go last night on my phone out of nothing more than wild curiosity. You can bet I learned a bunch about Go in the process, from bed, in about 20 minutes.
Generally I agree with your takes and find them very reasonable but in this case I think your deep experience might be coloring your views a bit.
LLMs can hurt less experienced engineers by keeping them from building an intuition for why things work a certain way, or why an alternative won't work (or conversely, why an unconventional approach might not only be possible, but very useful and valuable!).
I think problem solving is optimization in the face of constraints. Generally using LLMs IME, the more you're able to articulate and understand your constraints, and prescriptively guide the LLM towards something it's capable of doing, the more effective they are and the more maintainable their output is for you. So it really helps to know when to break the rules or to create/do something unconventional.
Another way to put it is that LLMs have commodified conventional software so learning when to break or challenge convention is going to be where most of the valuable work is going forward. And I think it's hard to actually do that unless you get into the weeds and battle/try things because you don't understand why they won't work. Sometimes they do
I think it's very easy to harm your learning by leaning into LLMs.
What I don't believe is that it HAS to be like this. Maybe it's my natural optimism showing through here, but I'm confident it's possible to accelerate rather than slow down your learning progress with LLMs, if you're thoughtful about how you apply them.
An open question for me is how feasible it is to teach people how to teach themselves effectively using this new technology.
I have a core belief that everything is learnable, if people are motivated to learn. I have no idea how to help instill that motivation in people who don't yet have it though!
2 replies →
What would you have us do, though?
Stifle the tools, somehow?
You’ve had nontechnical devs since npm, or before!
No: people that care to understand the whole stack, and be able to provide that value, will still exist and shine.
1 reply →
Agree completely. The other aspect for me is that LLMs make me unafraid to take on initiatives in areas I know nothing about and/or am uninterested in pursuing due to discrepancy in effort vs reward. As a result I end up doing more and learning more.
The struggle is how you learn. I think that’s pretty much established scientifically by now?
If it is I'd very much like to learn more about the science.
I find it hard to believe that wasting hours hunting for a missing semicolon (at the very real risk of quitting entirely) is essential for learning. Does that mean every student who asks a TA or fellow-student to help them find that semicolon is hurting themselves when they do that?
If not, what's different about asking an LLM?
5 replies →
You can spend 30 min, watching someone learning how to ski, you will learn something. You will not be able to ski by yourself though.
If you are not failing you are barely learning anything.
Yeah and then it becomes an unmaintainable monolith because at some point the AI also lost track of what code does what.
Great for Opus because you’re now a captive customer.
Yes, it's a risk if you don't guide it well, but you can also manage it pretty ok.
I have a side project that I started in January 2024. Initially, used Github Copilot autocompletions heavily. This year I started using CLI agents (mostly Claude, but others too) to do more stuff. I got to around 100k LoC (sure, it's not enterprise scale, but for a personal project it's pretty big), but I'd argue it's maintainable, it's split into 10 Django apps, that are each pretty self contained, I've done several refactors on it (using AI agents) to make it more maintainable.
The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from scratch, if needed.
I don't buy it.
I think that could work, but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature.
Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them.
I expect that we'll learn over time that LLM-managed big ball of mud codebases are less valuable than LLM-managed high quality well architected long-term maintained codebases.
11 replies →
It does matter because the code needs to still be legible and discoverable and semantic enough for other AI to find it and use it without it being so confusing or painful that they prefer to just write it themselves.
The reason software is so valuable is that it's capital/up-front investment in figuring something out that can continuously deliver value with low or no marginal cost. Rewriting/maintenance/difficulty/figuring out software is marginal cost.
Recreating everything from scratch gets harder and the previous requirements will eventually not be met after sufficient number of them have been accumulated. AI would have no solution to this unless it iterate on the same code base, but since I've not seen evidence of architectural maintainability from AI, a project that are fully given to AI is bound to fail.
AI is still incredibly useful used in tandem, but have it implement full feature from one sentence usually lead to doom.
In the case of OP, they cannot even test it, because they have no clue how it works. They cannot test whether the goal was achieved or not.
The other day I generated an MCP server for AST of Java. I had no clue how that works. I couldn’t test it because I had no idea how that looks like. Btw, AI even lied in tests, because it literally mocked out everything from live code. So everything was green, and literally nothing was tested, and it was untestable manually by me.
If you don’t know that Opus isn’t an entity, but a model,
you might be a little too far removed from the situation to comment authoritatively?
> I learned essentially nothing about building vector search. I wanted the feature more than I wanted to know how to build the feature
Opus/Anthropic is hands down the best in my experience. But using it feels like intellectual fast food (they all are), I hate the fact that I can build something like a neatly presentable one off spa tool (ty Simon) when I'm barely paying attention. it feels unsatisfying to use.
EDIT: because I'm rambling, I like "AI" as much as the next guy, probably more because I was there before it turned into LLMs"R"US, but I also like(d) the practice of sitting around listening to music solving problems with Scala. I don't know why we've decided to make work less fun..
“We” didn’t decide to make work less fun, others decided for us.
I sort of disagree. It's somewhat like having hypercard again. You can build fun UI things and make machines do what you want them to do. You can care about the parts you want to care about and not sweat about the parts you don't want to learn in detail (yet). And Claude and codex make great guides/Sherpas.
There are just too many parts involved to do anything. For example today I built a simple data collection app to use on my phone that involves inventories with photos for a tedious workflow I have to do. I knew what I wanted but didn't know how to even choose which tools to bother learn. And just even trying things to see if an approach works or not without spending hours learning one thing or another or wading through the hell of web search is really great.
Things I learned today that I figure everyone else must know: if you want to take a photo from a webapp I guess you need https. So I decided to try mTLS (knew it existed but never had the time) so asked Claude to write me a short tutorial about setting it up, creating keys, importing them (including a cool single line trick of spinning up a python server and downloading the keys on my phone rather than find a USB stick or whatever). And then helping me figure out a path out of the suffering of Chrome and Firefox hating self-signed CA. But at least I figured out how to make Firefox happy. But it would insist on prompting me for the certificate for every htmx request. But chatting with Claude I learn caddy is pretty cool, it's go. Claude suggests an auth boxcar when I balk at adding auth and user management to my app because I think the webserver should handle all this shit (wtf is a boxcar? Claude clues me in). I tell Claude to use go or rust to build the boxcar because Jesus Christ "yay" build another service just to get a good damn customized CRUD app on my phone that can take a picture. Claude picks go which is fine by me. (Incidentally I can't write go, but I can read it and it's on my "to be learned" agenda and go seems safer than a pile of python for this simple thing) The boxcar was fine but Claude was struggling with getting headers to work in the caddy config. So while Claude is working on that I do a quick Google about whether caddy can have extensions because there has to be a better way to "if someone has authenticated successfully, give them a cookie that will last an hour so they don't have to mash the confirm about using the certificate for every goddamn htmx request" than spin up a web service. Interrupt Claude and suggest an extension instead of a boxcar. Claude's on board so we ditch the boxcar. Have Claude and codex evaluate the extension for security. They find important issues about things a jerk might do, fix them. So successful mTLS connections transition to session cookies. So my dumb CRUD tool doesn't have to worry about auth. Which it didn't have to do anyway except browsers say so etc because my phone is literally only able to access the server via VPN anyway.
Other things I have learned today that only wasted 5min of Claude's time rather than hours of mine: Firefox camera access can't control flash, focus or zoom. So call out to the native app instead.
This is all quite fun and the tool I'm building is going to really make my own life better.
Is there a better way to do this: probably.
>only wasted 5min of Claude's time rather than hours of mine
I mean will you (we) retain all that it did after a few months go by? You may say we don't need to, but that sounds a little shallow given we're both on HN. Do you remember Gatsby's criticism of "Summer People"?
1 reply →
> I wanted the feature more than I wanted to know how to build the feature
This is exactly what LLMs are great for. For instance, I'm looking at trading models. I want to think about buying and selling. I need some charts to look at, but I'm not a chart wizard. I can make basic charts, but it feels tedious to actually learn the model of how the charting software works. LLM will just give me the chart code for the visualization I want, and if I ever care to learn about it, I have it in a form that is relevant to me, not the form of the API documents.
In general, a lot of coding is like this. You have some end goal in mind, but there's a bunch of little things that need to be knitted together, and the knitting used to take a lot of time.
I like to say the LLM has reduced my toil while getting me to the same place. I can even do multiple projects at once, only really applying myself where there is a decision to be made, and it's all possible because I'm not sorting out the minutiae of some incidental API.
I like having the flexibility. If it's something I want to learn, I'll ask it to write some explanation into an md that I can read, and I can also look at the code diff in more detail. but if it's tedious things like interacting with the android sdk, I'll just let it do whatever it needs to do to get the feature working.
Can we see that vector search code or use it?
The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it.
At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would have originally spent trying to understand it so you can fix it yourself.
At worst the thing will actively break other things you do understand in ways you don't understand, and you'll have to spend at least 4 hours cleaning up the mess.
Either way it's not clear you've saved any time at all.
Respectfully, I think I’m in a better position to decide a) what value this has to me and b) what I choose to learn vs just letting Opus deal with. You don’t have enough information to say if I’ve saved time because you don’t know what I’m doing or what my goals are.
Respectfully, a) I didn't say anything about what value this has to you but moreover...
b) you also don't have enough information to say if it's saved you time because the costs you will bear are in the future. Systems require maintenance, that's a fact you can't get rid of with AI. And often times, maintaining systems require more work than building them in the first place. Maintaining systems tends to require a deep understanding of how they work and the tradeoffs that were decided when they were built.
But you didn't build the thing, you didn't even design it as you left that up to Claude. That makes the AI the only thing on the planet that understands the system, but we know actually the AI doesn't understand anything at all. So no one understands the system you built, including the AI you used. And you expect that this whole process will have saved you time, while you play games?
I just don't see it working out that way, sorry. The artifact the AI spit out will eventually demand you pay the cost in time to understand it, or you will incur future costs for not understanding it as it fails to act as you expect. You'll pay either way in the end.
1 reply →
You do learn how to control claude code and architect/orient things around getting it to deliver what you want. That's a skill that is both new and possibly going to be part of how we work for a long time (but also overlaps with the work tech leads and managers do).
My proto+sqlite+mesh project recently hit the point where it's too big for Claude to maintain a consistent "mental model" of how eg search and the db schemas are supposed to be structured, kept taking hacky workarounds by going directly to a db at the storage layer instead of the API layer, etc. so I hit an insane amount of churn trying to get it to implement some of the features needed to get it production ready.
Here's the whackamole/insanity documented in git commit history: https://github.com/accretional/collector/compare/main...feat...
But now I know some new tricks and intuition for avoiding this situation going forward. Because I do understand the mental model behind what this is supposed to look like at its core, and I need to maintain some kind of human-friendly guard rails, I'm adding integration tests in a different repo and a README/project "constitution" that claude can't change but is accountable for maintaining, and configuring it to keep them in context while working on my project.
Kind of a microcosm of startups' reluctance to institute employee handbook/kpis/PRDs followed by resignation that they might truly be useful coordination tools.
I agree with this sentiment a lot. I find my experience matches this. It's not necessarily fast at first, but you learn lessons along the way that develop a new set of techniques and ways of approaching the problem that feel fundamental and important to have learnt.
My fun lesson this week was there's not a snowballs chance in hell GitHub Copilot can correctly update a Postman collection. I only realised there was a Postman MCP server after battling through that ordeal and eventually making all the tedious edits myself.
Yeah, this is close to my experience with it as well. The AI spits out some tutorial code and it works, and you think all your problems are solved. Then in working with the thing you start hitting problems you would have figured out if you had built the thing from scratch, so you have to start pulling it apart. Then you start realizing some troubling decisions the AI made and you have to patch them, but to do so you have to understand the architecture of the thing, requiring a deep dive into how it works.
At the end of the day, you've spent just as much time gaining the knowledge, but one way was inductive (building it from scratch) while the other is deductive (letting the AI build it and then tearing it apart). Is one better than the other? I don't know. But I don't think one saves more time than the other. The only way to save time is to allow the thing to work without any understanding of what it does.
> inevitably Claude reaches a point where it can't help.
Perhaps not. If LLMs keep getting better, more competent models can help him stay on top of it lol.
You're still captive to a product. Which means that when CloudCo. increases their monthly GenAI price from $50/mo. to $500/mo., you're losing your service or you're paying. By participating in the build process you're giving yourself a fighting chance.
1 reply →
Well, look through it's log and what it did and if you don't understand anything ask it why it did it/what it does.