One wrong advice clump and, like a step onto the wrong path while hiking, all subsequent steps go in the wrong direction. And sycophancy tuning means marginal one-sides takes get presented as sure-fire things.
I’m of the opinion that the big wins aren’t in using the LLMs to do the work (legal, in this case), but rather to refine and improve the dialog and presentation from all parties. A court-centric LLM that could give likely procedural needs to a litigant, and a law-firm-centric LLM could help a pro se litigant create a meaningful and refined set of questions for lawyer consideration, condensed and targeted, saving all parties time and confusion while meeting the clients linguistic needs ‘where they are’.
All the lawyers know things LLMs never will, the law is interpreted, and the written part isn’t engineering grade facts but suggestions interpreted in context. Arguably this is a racket and a thin veneer of plausible deniability for authoritarian rule. But as the law stands even with federal statues and citations from the courts website, practicing lawyers will frequently end up explaining that in this county/country/court/jurisdiction The Way of Things is different.