So if a chat program can pass the bar exam, it's okay? Because I would bet that if a program can represent someone semi-competently in court, passing the bar exam which needs an objective marking criterion would be trivial by comparison.
But a fun fact is that magistrates generally aren't required to pass the Bar, nor hold a law degree. Most states require extremely basic courses of 40 or so hours of training. I know of a magistrate that has tried numerous times to pass the Bar and has failed. I'm not sure how much competence our system mandates.