I rather think that they (the regulators) are saying that you can’t expect to use mathematical impossibility as a legal argument to avoid liability if your AI does material harm.
Or at least mean that if you're too poor for a doctor or lawyer you can't fall back to the unreliable AI?
Quantifying your risk does not mean you eliminated all your risk.
We allow the sale of alcohol, tobacco, and firearms too.