Comment by granfalloon
9 hours ago
As a white-collar worker, it really does seem like my programmer friends are living in a different reality when it comes to AI. They have totally restructured their workflows around AI, and I'm still struggling to find the value in it for my legal and compliance consulting work. (And I consider myself pretty tech savvy, with some programming and web development knowledge and having spent 10 years at crypto startup.)
Even in my personal life, while I've been addicted to using Claude and ChatGPT as a first step for research/projects/etc, I'm still not convinced they are actually saving me time -- just too many errors, too confidently stated, and I have to re-do all the research myself.
I wonder if the validation piece is easier with coding?
EDIT: To be clear, when I say I'm having trouble incorporating AI into my white-collar work, I'm talking about producing GOOD work. On a few occasions I have given in to the temptation of using Claude to produce work that looked good (superficially), without verifying everything.....it worked out, and I'm sure others are doing it. But that scares me, because it feels like a ticking timebomb.
Sizing the steps matter. Disclaimer up front: we are all figuring it out as we go along, there are no hard and fast rules that fully generalize. You just need to try and develop your own feel for it.
In my case, for some types of work - which sound similar to things you mention - I started slowing AI down intentionally. Software development, if it's something simple/common and thus "in-distribution", LLMs can zero-shot complex projects just fine. But in more R&D work, I found that errors crop up quickly and accumulate, and end result of "go and just make X" is usually bad.
Now, the problem I found with usual workflows for this kind of work is, agents race ahead and very quickly, I can't keep up anymore. So I reworked my workflow to make AI work in small steps optimized strictly for maximizing my ability to validate the outcome, and minimize time I spend doing it. So now, the thing (say diagnosing a performance regression of some complex codebase run on a cluster) may take me a whole day, where "unchained", agentic swarm would do it in 30-60 minutes - but, at the end of the day, my method is near 100% guaranteed to be correct, with every intermediate step vouched and reusable for future work, whereas "agentic swarm" approach, that hour of work, 90% of the times I'd just skim and throw away and reprompt, because I couldn't even begin to guess which part is correct, which isn't, so in the end, it was just a fast way to waste lots of tokens.
I see many people have complex systems involving skills, agent swarms with roles, agent factories, 20 termux panes and whatnot. Maybe that's the way, I don't know. Nobody knows. But it's not necessary to start getting value from LLMs. I therefore recommend my "lazy person's basic approach":
- Throw a task at LLM
- Check result, see what you like or not like about it
- If you just can't keep up, ask it to do smaller steps; if you don't understand, ask it to explain
- Just do that a lot, and don't be afraid to make both absurdly large and absurdly small asks
- You'll quickly get a feel for where AI can help you, and where it's not yet worth the time handholding it
Thanks, I'll continue to experiment!
I've used LLMs for some very small coding and math tasks, and I've found it much easier to work with on "formal" types of problems.
But for my consulting and legal work, which is "fuzzier," I struggle more with verification. I feel like I have to verify every single fact/premise in the LLM's output (this problem doesn't seem as big with coding?), as well as every single logical conclusion. Plus, even with explicit instructions I feel like the LLM is not great at anticipating what certain audiences will be interested in or find compelling.
Compared to the black-and-white nature of a computer program, all the components of a legal memorandum (premises, logical structure, conclusions, etc) seem so much fuzzier and require so much specific and accumulated context to apply reliably.
Really well put. This parallels my approach and it's good to see someone advocating a more tactful use of LLMs.
The output of agent swarms will of course grab attention, just like when you saw those same Microsoft Frontpage themes in the first few years it came out. But there's a deeper value in using the tool in harmony with your own experience and value additions.
Opus 5.5 is superhuman at most white collar work. At this point, not using it (including for health/law) is basically malpractice.
"Superhuman" in what sense? With respect to legal work.
Is it just faster? More persuasive? Better at coming up with novel arguments?
Speed is great from a client perspective, but not as important as accuracy on important stuff. I haven't seen LLMs produce arguments that are more persuasive than those crafted by good lawyers. Finding the "perfect case" in Westlaw is helpful, but rarely the deciding factor.
I could see asking an LLM to reproduce your work AFTER you've done a first pass, to see if you missed anything -- that seems like a real value add. But I truly can't understand how not using it would be "malpractice." In fact, I still think committing malpractice would be MUCH more likely with AI. (Now if by "malpractice", you meant pissing off clients with expensive bills, that's a different story...)
I think the parent post meant that it's very good at writing code and operating the computer. I think we're also biased because we've used older generation models and the latest ones are orders of magnitude better than what we had 6 months ago.
For example, for your legal work, it can:
Develop a machine learning model (CRF, running on CPU) for clause extraction from scratch, Find training documents and training data and train the model for various provisions (you might help it here by pointing it to the directory with annotated documents),
Build an app where you upload your legal documents and it extracts the provisions you want from all documents.
Integrate semantic search and an AI agent into your app which lets it answer any question about your documents with exact evidence from the documents.
All of this from one well crafted prompt, a few skills and a workflow. It could probably complete this in one night, while you sleep.
Or you can point it to your saas of choice and tell it - 'create me a copy of this app'.
All you need to know is how to steer it through the planning/research phase.
So it's not that it's 'accurate' in legal terms, it's that it can build you any tool you want from scratch so that you can become more accurate.
I'm not in the legal world, so my experience is limited from a coding perspective. I don't think AI could ever truly come up with novel arguments, at least not yet, but their value for you may be to sharpen your arguments or pull on every thread that you may not even have thought about.
For example, theres a very popular coding skill set call Matt Pocock's Grill Me where it just asks you a relentless amount of questions about the feature you're trying to build. You may be able to tailor this somehow to act as the opposing counsel and have it poke holes in your argument. I suppose the closest analog would be a sparring partner.
Its helpful for me and my ADHD brain because I can go down all sorts of rabbit holes and it can easily keep track of all the ideas i have while grilling and it can send sub-agents out to investigate other claims to add more context to the grilling session.
I've found it depends a lot on how Claude is setup, and spending some real time on the default Private Instructions to focus on accuracy, identifying/flagging uncertainty, citing sources, checking itself, and first surfacing all assumptions and asking questions dramatically pays off in terms of getting concise and solid usable results.
Edit: add: I've found this to the degree that a relative who works at a top biglaw firm and has their internal AI (MS-based) and used to find ChatGPT and Claude actively time-wasteful, when I posed a question they'd spent a day+ fruitlessly researching (multiple areas of law and international), Claude came up with a very useful answer in ~10min — they were quite impressed.
The last year has made a huge difference