As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.
Opus 5.5 scores 4% on it.
Imagine a bunch of lawyers arguing what programming is. Even a bunch of programmers arguing what programming is abnoxious at worst and a mental exercise at best.
Domain knowledge is informative sometimes but mainly overrated: the success of LLMs proves just that. What most lawyers do is fairly standardized and routine, efforts to invoke mystery cases or "novel" is an attempt at resisting the inevitable. The "the only British food in Palo Alto" equivalent of unique skills.
True of...literally every occupation.
If lawyerin’ was always doing something novel, it wouldn’t be much of a set of laws. The bulk of lawyerin’ in the US is, I would guess without a whit of substantiation, a lot of small claims and standard form type things. Not discounting the value of the profession in any way, but if one job based on standard codes can have its basic functions replaced by AI, why not another?
Programmers typically don’t know law outside of which license allows them to get code for free.
Yeah, and how many of drivers or software engineers can fix their car's engine? Just a few. Even though in priciple there's not much to it.
b) I have met Lawyer colleagues who didnt know how build a =SUM() in Excel
So, I doubt....
That doesn't mean they don't exist. I went to law school with many former software engineers, many of whom are now prosecuting patents on behalf of high tech companies, and some of whom went back to engineering (like me).
"I have never met ______" is of no consequence to anyone; it's an argument style best avoided unless you're talking about deities or space aliens.
See https://hls.harvard.edu/bernard-koteen-office-of-public-inte...
How much of your work is truly novel and unique? Not 100%
An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases.
Look at the trajectory not where it’s now.
As much novelty as there is in human interactions.
Infinite
Makes sense.
If you don't have that, look at how many people bitch that Google or Paypal or whatever cut them off with no warning and no concern for the circumstances. Now imagine that, but like for your house or your kids.
Law is not clear, rational code. Law is human drama, in all its many, many forms.
People like to pretend “CRUD” is this solved problem and I see it is, in theory, yet in practice CRUD hides a lot of bullshit that gets swept under the rug.
In a very interesting way the systems I worked on were always encodings of “centuries” of drama, psychology and politics. It just happens a lot faster on the software side.
I was allowing it for the sake of argument, because, yeah, any app of any reasonable size is not simple.
Law looks like rules, but it's not. It's centuries of codified human drama and fighting. It's mostly edge cases, because you really only fight when something goes wrong.
On top of that, lawyers are vehicles for liability. Can't give that to a chatbot lol
I don’t see the problem to be honest. Edge cases don’t bother the clankers. You think “programming”, all of it across all domains, plus the business side of things (which is a source of insanity itself), the pointy haired bosses, the ridiculous processes, the “UIs”, the complexity of a full system from top to bottom is rule-based and orderly?
Myopically looking at dev work makes it seem like this abstract wonderland but it’s seething, bristling whirlpools of psychology, neurodivergence and misaligned stakeholders all the way down.
Your point on liability is fair. I guess we can appoint liability crumple zones but I suspect they won’t rake in 200-300 an hour. Besides liability can very much be given to a “chatbot”. I don’t see how any legal entity can’t take it on. I agree that society is not ready for it, not at all, but I fail to see the fundamental blockers.
Law is extremely unforgiving. In general, you get one shot at it, and if you lose, that's it. No do overs, and the consequences can be life-altering. That's why so many things in law are the way they are, and why it's illegal to practice law without a license.
Because the stakes are so high, lawyers are also vehicles for liability. If you ask one for advice and you lose because that advice was wrong, you can at least recoup damages from them.
You lose all those protections with a chatbot.
All of that is testable if the incentives are there and AI incentivizes testing.
Good luck testing law!
EDIT: specifically around the use of judicial precedent in case-law.
90% of legal work where there is a plaintiff and defendant is pressuring a settlement. My wife, an attorney, does not consider that to be "litigation".
But, for your sake, it's probably best that you accept your wife's definition.