AI can filter through thousands of Legal case files and court documents to pinpoint exactly what cases are relevant to a current case that a lawyer may be advising on! It took me exactly 30 minutes to go through 428 case files... :)
My experience so far has just been asking chatgpt questions and then researching it myself to confirm what it says, so maybe I'm missing something. But, it has been confidently wrong on important details a large enough percentage (right now) to make it absolutely not a fire and forget tool.
The worst part is the confidence: it's like having a coworker that just straight up lies to your face randomly. Even if it's only 5% of the time you basically can't trust anything they say, and so you need to double check all of it.
This doesn't make it useless, but it means it lends itself to "hard to do but easy to verify" tasks. Which afaict your example is not: you can verify the documents it picked out are relevant, but not that the documents that it didn't, weren't.
* their reputation with respect to the question domain (if I ask a basic C++ question to a C++ expert I'll trust them)
* their own communicated confidence and how good they are at seeing their own shortcomings (if they say "but don't quote me on that, better ask this other person who knows more" it's fine)
5% of bad answers doesn't matter if 99% of these times I knew I should look further. ChatGPT and others are missing this confidence indicator, and they seem to answer just as confidently no matter what.
To be clear I don't see a fundamental reason why LLMs couldn't compute some measure of confidence (which will itself be wrong from time to time but with less impact) so I expect this to be solved eventually.
It's a huge boon, but nothing is perfect. If it's that important that absolutely nothing is missed, maybe try multiple approaches concurrently. Take this as a value add, not a replacement.
I'm not a lawyer, so when I search British copyright law, I wonder why those forms of words don't result in the staff of search engines and social media sites being arrested and imprisoned:
https://www.gov.uk/government/publications/intellectual-prop...
Although, probably best to ask a lawyer for legal advice even if they end up using GPT-n themselves; they've probably got public liability insurance if they get something wrong.
AI can figure out from context, a keyword search will miss it. And no, you don't know the "code words".