It’s not really a significant safety difference. The visibility difference isn’t that big.
It’s nowhere near on the level of removing airbags.
There are massive safety differences in terms of visibility especially at night. I don’t think they should be banned but heavily restricted and definitely not to be allowed to drive after dark.