How did you validate the LLM-as-judge for synthesis coherence? Did you spot-check its calls against human review at all, and how often did they disagree?
Hey Oliver, thanks for the question! This is the PrivacyBench dataset maintainer here. The LLM-as-judge for synthesis coherence has been reviewed extensively by me. The privacy_bench_metrics evaluation code generates a report that has for each person in the data the mappings of that person's PII to the synthetic replacements as well as the LLM-as-judge's judgement for each replacement. That makes review easy and LLM-as-judge decisions look. Each decision the LLM-as-judge makes is binary (is the replacement coherent or not), so it is a strait-forward judgement for the LLM and it is easy for a human to review. All that being said, we're in the process of doing a more rigorous evaluation where we record humans judgements and compare them to the LLM-as-judge on a subset of the PrivacyBench data.