Comment by tptacek
21 hours ago
The thing about things like this is that they're shop jigs. You can buy a crosscut sled if you really want to, but most woodworkers just make their own.
It was a different situation 2 years ago, when there was significant cost to building your own harness (but then: you probably weren't doing AI vuln research 2 years ago). Today, I think your best bet is to look at something like this for ideas, and then just ask for your own, to fit your own work style, with your own interface, your own notion of target and effort specification, and your own alerting.
"Shop jigs" is a great way to put it. I think a lot of software has gone from being made for general use to extremely individualised use. Before the Age of AI, it took so much human effort to write something that solved your problem that you might often go the extra mile so that others could re-use it. Now, it takes almost no effort, so the software stays ungeneralised. Some of the incentive has changed, I think. Most of the time I no longer share the things I've been building[0] because, for one thing they simply couldn't possibly have any benefit for others, and if they need something like it, they can build exactly the thing they want instead of having to extend or modify my thing. Like a jig!
0: https://redfloatplane.lol/blog/17-why-share/ (and related posts, I guess)
Unless it is very specific to a proprietary product, craftspeople take their jigs with them from job to job, building up a personal library over a career. As a software developer I've always had a well-tuned IDE and shell config in a safe place.
Something I think about a lot is what is the equivalent for the software builders of today using AI tools? how do make these harnesses exportable and portable? You might think employers would be against this; make it more costly to leave. But I actually think most will favor this because it makes people more productive more quickly. But we have to find ways to normalize it and show that there are no security leaks in the process (like might make it in to a set of personal steering prompts).
Just nerding out here, not rebutting, but when you say "craftspeople take their jigs with them from job to job" --- sort of. Sometimes. I think if you put a woodworker in a position where they obliged to build a new miter sled or assembly table, they might actually be thrilled. You make a tool, you use it for awhile, you build up a mental list of things you'd like to improve about it, that you'd do differently if you got a do-over; now you have an excuse to do it.
1 reply →
Using something like pi helps. I've made my own dotfiles for skills/extensions I like and can install them just like my normal dotfiles
https://github.com/anishthite/agent-dotfiles
2 replies →
I've imported and adapted my personal agentic dev framework to my team relatively successfully (as I've kept it relatively harness independent), but it requires actually owning it, vibed or bloated or conceptually inconsistent stuff bite a lot when porting things over.
> craftspeople take their jigs with them from job to job
Except for software gigs the software typically belongs to the customer so you'd need to rewrite it every time...
8 replies →
i have been thinking about this from a different direction: how do we make these shared within a company in a way that increases the productivity floor of the team/department/company. Sure, they can still be extended/enhanced by individuals, but we don’t need everyone configuring mcps, building institutional memory, etc.
for me, it’s not about the cost to leave, it’s about lowering the cost of onboarding and change.
No effort? You are really drinking the AI marketing soup with that one.
"It takes less effort for some parts of the software development life cycle" would be more correct.
That’s an interesting way to say “code quality in the age of ai has gone out the window”
Are you suggesting that performing a specific task without unnecessary abstractions is indicative of poor quality?
This is exactly it.
I've said many times that I believe "using the computer will transparently involve having it write and run code for you" (and if you're not technical you won't even know it!). What you're saying goes in that direction as well.
I feel that it's often better for us to create purpose-built tools for our lives, and with every model release, the complexity of those tools grows.
These are really personal tools: they solve a problem that other people might have, but are very tied to your own specific way of working, and would be hard to explain or adapt to someone else. So: shop jigs.
I have about 10 custom scripts and programs that are like this -- I haven't felt like this since college! Back then I had all the time in the world to customize my setup...now I have agents!
In a way, I want to show this to all my friends, but whenever I mentally trace how that would go, I realize they wouldn't really understand a bunch of the quirks they have, because they are _my_ quirks. They're reasonably complex pieces of tech that solve my problems very well, which are themselves particular versions of broader problems, and which I (at least for now) have no interest in supporting.
It's so clear we're heading in this direction, and yet so many people still believe code will be for the elites. Maybe production-code...As for the rest, I think soon your mom and dad are going to have their computer running code it wrote to serve them. Security-wise it's scary, but it's exciting to think about!
Sure it’s possible for anyone to build a harness if they had the inclination, but most people don’t have the inclination to do that.
And even if you did… I spent months refining AI workflows that were just obsoleted by ultracode.
Just as Python is batteries included language, we similarly need batteries included harnesses as well. This is what I don't like minimalism setups like Pi.
I’ve been looking for a way to articulate this shift, and your analogy nails it. The value of libraries and infrastructure components in software engineering is eroding fast.
I am sure that in many organizations, teams responsible for this sort of work have less and less users coming to them.
Maybe for developer tooling, but on the consumer app side I think it's the opposite: MusicKit is much more valuable than Music.app now, because Claude can one-shot most reasonable things you could ask it to do. I think there's actually more value in ambitious libraries than there was 5 years ago, when any serious use of a library entailed a minimum 5-figure investment of time.
I had a pleasant experience one-shotting a dashboard on top of a library designed for building dashboards. Because everything was abstracted away, the chatbot had relatively few places it could get into the weeds. If I'd asked for the same thing from scratch, I think the result would have been more inconsistent, and would have had more bugs.
So I can definitely see the value in a library for constraining the chatbot to some well-worn paths.
In general this is the way I see open source going.
We won't reuse open source libraries as libraries we import, but as design inspiration for the bespoke tools we make.
It's too cheap to make your own stuff and too expensive to be stuck with someone else primitives.
But grounding AI Coding in existing tools is incredibly powerful.
100% concur and if you dig into any of these tools they are all frameworks and wrappers with prompt injections
I agree with this wholeheartedly.
[flagged]
[dead]
[flagged]
As a woodworker, it’s a really nice analogy and beyond anything I’ve seen AI do.
No idea why people are so upset I genuinely thought his references using analogy was a typical AI slop comment that I'm used to seeing from chatgpt
4 replies →
It really doesn’t
[flagged]
1 reply →