Comment by TeMPOraL
8 hours ago
Sizing the steps matter. Disclaimer up front: we are all figuring it out as we go along, there are no hard and fast rules that fully generalize. You just need to try and develop your own feel for it.
In my case, for some types of work - which sound similar to things you mention - I started slowing AI down intentionally. Software development, if it's something simple/common and thus "in-distribution", LLMs can zero-shot complex projects just fine. But in more R&D work, I found that errors crop up quickly and accumulate, and end result of "go and just make X" is usually bad.
Now, the problem I found with usual workflows for this kind of work is, agents race ahead and very quickly, I can't keep up anymore. So I reworked my workflow to make AI work in small steps optimized strictly for maximizing my ability to validate the outcome, and minimize time I spend doing it. So now, the thing (say diagnosing a performance regression of some complex codebase run on a cluster) may take me a whole day, where "unchained", agentic swarm would do it in 30-60 minutes - but, at the end of the day, my method is near 100% guaranteed to be correct, with every intermediate step vouched and reusable for future work, whereas "agentic swarm" approach, that hour of work, 90% of the times I'd just skim and throw away and reprompt, because I couldn't even begin to guess which part is correct, which isn't, so in the end, it was just a fast way to waste lots of tokens.
I see many people have complex systems involving skills, agent swarms with roles, agent factories, 20 termux panes and whatnot. Maybe that's the way, I don't know. Nobody knows. But it's not necessary to start getting value from LLMs. I therefore recommend my "lazy person's basic approach":
- Throw a task at LLM
- Check result, see what you like or not like about it
- If you just can't keep up, ask it to do smaller steps; if you don't understand, ask it to explain
- Just do that a lot, and don't be afraid to make both absurdly large and absurdly small asks
- You'll quickly get a feel for where AI can help you, and where it's not yet worth the time handholding it
Thanks, I'll continue to experiment!
I've used LLMs for some very small coding and math tasks, and I've found it much easier to work with on "formal" types of problems.
But for my consulting and legal work, which is "fuzzier," I struggle more with verification. I feel like I have to verify every single fact/premise in the LLM's output (this problem doesn't seem as big with coding?), as well as every single logical conclusion. Plus, even with explicit instructions I feel like the LLM is not great at anticipating what certain audiences will be interested in or find compelling.
Compared to the black-and-white nature of a computer program, all the components of a legal memorandum (premises, logical structure, conclusions, etc) seem so much fuzzier and require so much specific and accumulated context to apply reliably.
Really well put. This parallels my approach and it's good to see someone advocating a more tactful use of LLMs.
The output of agent swarms will of course grab attention, just like when you saw those same Microsoft Frontpage themes in the first few years it came out. But there's a deeper value in using the tool in harmony with your own experience and value additions.