Comment by otterley
9 hours ago
As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.
9 hours ago
As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.
To be fair to them, on this particular benchmark (Harvey's Legal Agent) 20% puts it almost top of the pack of _any_ model, and certainly top for any remotely in its price class.
Opus 5.5 scores 4% on it.
AI is coming for the transactional work long before it will come for litigation. In 50 years, nobody is going to have a human write their will, for instance.
You surely mean <1 year. Hell I'd probably trust Fable/Astra right now and save myself the expense.
Writing a will isn't really "transactional", there is no negotiation involved. And wills are already a highly commoditised legal product. (So I agree with you that will drafting is ripe for disruption by LLMs, but don't think this indicates a broader disruption of transactional work or the profession generally.)
50 years is way too long a time horizon to predict much of anything
In 50 years time 95% of legal complaints will be decided by AI
It is amusing how many people are arguing what litigation is here, probably none of these people have a law education.
Imagine a bunch of lawyers arguing what programming is. Even a bunch of programmers arguing what programming is abnoxious at worst and a mental exercise at best.
How does that track though? If, as a programmer, I claim programming is really the tasks a principle engineer does at Google or Netflix, is that accurate or survivorship bias from someone who made it to the top of the chain?
If lawyerin’ was always doing something novel, it wouldn’t be much of a set of laws. The bulk of lawyerin’ in the US is, I would guess without a whit of substantiation, a lot of small claims and standard form type things. Not discounting the value of the profession in any way, but if one job based on standard codes can have its basic functions replaced by AI, why not another?
That's how I feel reading HN's opinions on the recent math results. <5% of the responses even understand the point they're supposed to be arguing. AI might take over mathematics. It might not. The discussion regarding that topic is almost devoid of any relevant information though.
I'd extend that to the typical HNer's understanding about practically any subject. A lot of bluster and supposition, little first-hand knowledge or expertise.
> probably none of these people have a law education.
Domain knowledge is informative sometimes but mainly overrated: the success of LLMs proves just that. What most lawyers do is fairly standardized and routine, efforts to invoke mystery cases or "novel" is an attempt at resisting the inevitable. The "the only British food in Palo Alto" equivalent of unique skills.
>What most lawyers do is fairly standardized and routine
True of...literally every occupation.
Lawyers would be better suited to argue what programming is because they know logic. There’s not much to algorithms: do this, then do that.
Programmers typically don’t know law outside of which license allows them to get code for free.
> There’s not much to algorithms: do this, then do that.
Yeah, and how many of drivers or software engineers can fix their car's engine? Just a few. Even though in priciple there's not much to it.
1 reply →
a) I have never met a Lawyer who was able to write a single code line
b) I have met Lawyer colleagues who didnt know how build a =SUM() in Excel
So, I doubt....
3 replies →
Objectively a 20% pass rate is not impressive. Yet, there is still utility to AI research and drafting.
Wouldn't be surprised, if a human attorney would get 5%, while copy-pasting ChatGPT answers, report in the results a week past the deadline, with a $12,500 retainer charged upfront.
If professional AI can do all the work that isn’t novel, you free up human talent for edge cases and then improve your AI with new information.
How much of your work is truly novel and unique? Not 100%
Totally fair. However, even if the LLM would come to the same conclusion and produce the same content that a competent attorney would, the attorney must still thoroughly vet the work for accuracy. So there is potentially time saved in research and drafting, but it’s not the miracle some might characterize it as.
An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases.
And fixing a broken draft can be more work than writing a proper draft in the first place.
What are you talking about? 90% of litigation is convincing the judge that existing case law that clearly applies to your case does apply to your case.
I'm guessing what they're saying is that nothing goes to litigation if it's a clear pattern-match to existing cases, so actual litigation practice is only novel cases.
Isn't that just in the US/Commonwealth judicial system? I could be completely wrong on how others, like non-adversarial judicial systems, work and/or how pervasive the US system is or is not.
EDIT: specifically around the use of judicial precedent in case-law.
Look at all these benighted folks who believe that American judges adhere to precedent when it’s brought to their attention!
90% of litigation is pressuring a settlement.
> 90% of litigation is pressuring a settlement.
90% of legal work where there is a plaintiff and defendant is pressuring a settlement. My wife, an attorney, does not consider that to be "litigation".
1 reply →
And most of the leverage you have in negotiating a settlement is by referencing prior case law. If something is novel, there's too much variance to get a good settlement.
2 replies →
[flagged]
How much novelty can there be? If attorneys were as diligent and generous with “open sourcing” their data as SWEs had been, could LLMs benefit from all that training data?
> How much novelty can there be?
As much novelty as there is in human interactions.
> How much novelty can there be?
Infinite
In the same sense there could be infinite CRUD apps?
5 replies →