A sound, centuries-old concept that pre-dates the modern justice system.
(Also, in most cases, you're wrong. Intent does not matter more. Which is why actions, not opinions, constitute the illegal activity. You can hate gay people as long as you make them their cakes. You can freely express Nazi ideologies as long as you don't plow your car into people. You can hate your wife as long as you don't murder her. You can hate your boss as long as you don't burn down the office building. Etc.)
> but you will surely hit some where many reasonable people strongly disagree after enough such cases.
Certainly. SCOTUS often passes down tests, but they can be very open-ended and sometimes even depend upon cultural context (e.g., Tinker). Also, those tests are sometimes updated as new cases are presented and the boundaries drawn by the previous test are judged insufficient (again, e.g., Tinker).
> This is bad because we're all biased in various ways and knowing about that bias actually tends to make it worse among the people who don't realize that simply knowing your biases doesn't make you immune to them, leading them to overconfidence.
How so? Judges understand that the law is full of grey area. When controversial issues of constitutional law come up, they often spend a great time deliberating. This is one reason why it's imperative to have a high-quality and independent judiciary. Our judges are not perfect, and some are even quite fallible. But this sort of self-assuredness in the outcome of dicey constitutional law cases is far more common from people off the bench than people on the bench.
Even on the problem at hand, it's pretty obvious that both of the "no grey area" positions are ridiculously untenable.
For the general case, what's your proposed solution? A system of laws with no grey area? I conjecture that this problem is even harder than AI-strong. Way harder.
We're just barely learning how to write bug-free device drivers in a formal language. What makes you so confident that we could solve all the problems of politics and philosophy, in a completely unambiguous way, in an informal language?