Show HN: Testing AI for Legal Document Classification

Hey HN!

I recently had the chance to experiment with ChatGPT to review legal documents for a client. They wanted to speed up the process of categorizing legal documents and wondered if AI could help.

I've been playing with ChatGPT for a few years and following all the nuances and challenges associated with LLMs, so I advised them to tread carefully. They agreed to a systematic test case to find the best approach and to keep law students and professors in the loop for the final QC.

I carefully developed and revised a pre-prompt to accompany each document via ChatGPT's Assistant tool. This prompt included relevant definitions for elections, document types, and descriptions for each tag. After the results were returned, I analyzed and summarized the findings and provided the data to the client for careful review.

The results were pretty promising!

They reviewed the same documents and came to the same conclusion as the AI the vast majority of the time! Additionally, the tags provided were accurate enough to help speed up the legal evaluation.

tl;dr - here are some other high-level results:

1. Both budget (4o-mini) and full-size (4o) models do a reasonable job determining whether a document is related to elections.

2. The newer, larger models perform adequately for the more complicated task of assigning tags to documents.

3. The mini model struggled to return accurate or relevant document topic tags.

4. By evaluating each document multiple times, we could check for "concurrence" of the AI's evaluation.

5. Cost is a factor, given the length of some of these documents. I estimated the backlog of documents at ~150M tokens. If we evaluate each document multiple times, the costs really add up!