Comment by jedberg
7 hours ago
I would love to see these benchmarks split up across "generic" tasks and novel tasks.
For example, redlining an OEM agreement with my vendor. They pretty much all use the same template, and the objections are always the same. I tried using Claude first, and even my lawyer was impressed that it found things they missed. Then I used Claude to redline that type of contract and the other side said "your lawyer was surprisingly thorough". I had my lawyer do the final check, but saved a bunch of money doing the first three rounds with Claude instead. And my lawyer basically just said "looks good to me" by the time they reviewed it.
Or even writing a simple will or estate document.
It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.
> It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal
This seems to be the case with most industries that LLMs are impacting, coding, data input, legal etc.
Whilst it's great to get the time saving from not delegating to a junior (LLMs are much faster at the basic level 1 work) there's going to be this big knowledge gap that you really only gain from actually doing the mundane boring shit.
A lot of this _should_ be automated away, but we really need to think about how the next group of people are going to learn the skills and nuance to applying them. I don't really have an answer to that one other than still giving that work to junior staff, but they will then just utilise an LLM to help complete the job, and rightfully so, I'd have done exactly the same thing.
I think OpenAI tried doing this with some early chatgpt settings with "learning mode" or whatever that is. But an LLM just telling you what to do will never teach someone, there was something about having to go out and hunt for the answer through the sea of crap responses or wrong SO questions that really taught you something. How do you replicate that learning with an LLM and how do you apply that to different industries like Law?
> but we really need to think about how the next group of people are going to learn the skills and nuance to applying them
That's what school is for. For the last 150 years we've automated a bunch of things, and then we send people to school to learn how those automations work, so they can make better automations.
Maybe law school will have to be a year longer. Or maybe some forward thinking firms will have new associates check the work of the LLM to gain that experience.
> That's what school is for.
It's not.
We have advanced CNC mills and lathes, and apprentices still learn doing things manually to understand how the tools work, their failure modes, and more.
There are machines that mix drinks, but you still learn to bartend by doing it manually and making mistakes, not starting out with machines.
Same with writing. Same with software. Doing the "simple", automatable thing is how you learn how the system works.
School can't be the same. Plenty of people have tried, but being effective in the real world is different.
In theory, theory and practice are the same. In practice, they're not.
1 reply →
In the future, I think AI lawyers will be arguing with AI lawyers for the most part.
Like how people use AI to write their resume and then the HR department uses AI to screen the resumes, the 2 AI's duke it out.
3 replies →
Still, school isn't quite the same as learning skills through work experience, is it?
> Maybe law school will have to be a year longer.
Law school is already 3 years long in the US (after 4 years of undergrad), and most agree the second two are a complete waste of time. Other jurisdictions are different, but the US market is by far the largest.
1 reply →
True, but have you seen the state of education in the year of our Ai-ssiah, 2026?
This is kind of a new complaint. Places like adp have being trying to automate away mundane hr tasks. Oracle and SAP have been trying to automate away the mundane admin tasks. Siemens does it for OT. There's a huge long tail of trying to solve this as if it were a problem in almost every field. This has been going on for decades and no one complains that we will lose expertise in these fields if the drudge work goes away. But bring in an AI... and everyone is suddenly concerned about corporate drudgery.
Anyone who does not do the grunt work of learning how things work at a granular level will run into trouble when their agent suggests "let's re-write the firmware!" and the user says "sure, whatever that means", the model will get stuck or break something, and there will be no one around to understand what is going on.
With strict version history that may be okay, but hopefully their agent already thought of that for them too!
Whenever you "automate away" things that are actually a part of you, you lose a part of yourself by definition.
When you automate physical movement, you atrophy. That implies ill health and helplessness/lack of resilience should your automata fail.
When you automate intellectual movement, the consequences are even more dire.
Most people lead a very immobile life already, in both aspects. They don't realize the consequences, if everybody did the same.
I don't think toiling away at mundane tasks is the only way to build expertise in a field, or even a particularly good one. Today's farmers barely touch hand tools anymore and they're able to produce crops just fine.
I've been having an identity crisis since generative AI started that is basically, "how am I going to learn anything with my ADHD brain when AI can do 10% of it immediately". That initial 10% would take me weeks or months to achieve on my own, and AI can do it right now, and that makes it even harder for me to bother. I hate it. I badly wish to do it myself because then I could achieve more than 10%, but now AI is right there and I just can't seem to justify the effort...
I did much the same with a placement agency. Even uninformed about the law beyond the handful of things I worry about, it became clear the company had simply welded a bunch of different boilerplate, from different countries, without actually putting a real lawyer in the mix. Or just having a junior look at it.
Weirdly, I wonder the net effect of me helping them accidentally tighten their terms to something enforceable.
Hmm, Claude, make a note to just figure out if the doc holds together before I bother.
> And my lawyer basically just said "looks good to me" by the time they reviewed it.
Last year when i mentioned this application of LLMs somewhere on Reddit (in the course of arguing that white -collar workers are an endangered species) it was met with scorn by commenters who insisted that machines could never match the skill and care needed to write and review contracts and the like, at least not to the level demanded by the legal profession. Well maybe the human standard isn't that high to begin with; my dad was just telling me last night the story of how his lawyer had urged him to cave to a plaintiff's demands because the judge who was hearing the case "gets bored and doesn't like reading the documents." (The judge's son is currently running for the office of county prosecutor, lol.)
The expected next step then would be for such judges to use AI to read the documents.
You then get a decision that your AI agent explains to you.
What could possibly go wrong? You will have to ask your agent for that.
It's always been a silly argument that's also predicated on a massive fallacy which everybody is aware of but seem to conveniently ignore during these arguments: that humans never make mistakes.
On top of that, dealing with text is literally what LLMs are all about.
The problem with accuracy, hallucinations and just generally ensuring responses are grounded in factual stuff is in large part and engineering problem.
Machines can’t be held accountable, which is problematic in law particularly.
I've spent quite a lot of time with corporate counsels over the past dozen years and one of my consistent findings is that they -- except in areas of liability and intellectual property -- almost always defer to the sponsoring business executive in defining terms and specifying the details of the contract. They are present primarily to keep biz leaders and their counterparties operating within an acceptable risk profile but usually don't have strong opinions on much of anything else.
So, to me it isn't curious that your experience with Claude has been positive. Provided you know what you're doing and you aren't asking Claude to go too far outside the expected norm, there wouldn't typically be a compelling reason for most contract attorneys to push back.
Where I work, we're trying to let Legal use AI for the first pass of things for like financial "onboarding docs" we negotiate with clients. Some tweaks the client wants we probably don't care about, some need careful review by a proper lawyer. It would be great if we could run it, and then on the ones with only minor tweaks we don't care about get the doc signed immediately.
I'd worry that professionals have learned to look for certain problems, and the LLM may sometimes be keeping the problem while transmuting its signature to evade detection.
Kind of like how they can take crazy ideas and dress them in better vocabulary.
so they eliminate your talent pipeline and cut down on billable hours? is this good?
Not good for the law firms, but great for the rest of us!
same amount of work, more capability per person. More competition, lower wages
I want to minimize billable hours, yes. Lawyers are insanely expensive, to the point of being inaccessible to most.
A family member works as a paralegal, and they have way more work than they can deal with. Their estate people have 70 cases per person that are mostly waiting on people to have time to work on them. Speeding this up can mean more billable hours quicker, depending on how the speedup works, and then they can just take on more cases.
how much 'novel' tasks are there in law?
The Legal Department at my work did not have the same success you did.
Every AI tool they tested completely shit the bed and ended up requiring more time for contract review than a human review, and completely failed to spot any of the issues that were actually relevant to our industry. The AIs were also for some reason really focused on lack of consideration, even though that was not an issue with any of the tested contracts.
When did they test them? Models have a different level of capability than 6 months ago, IMO.
> It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.
This seems to be the common thread everywhere LLMs end up being useful. As the sole software developer? Pure trash, but as "tons of juniors we can shuffle boring tasks at": great. As the sole designer and maintainer of a website? It'll look like 150% slop and no will buy anything, but as a designers friend to create what they dictate: even a designer can throw together a good looking website quickly. Etc etc.