Seems really shortsighted view. What about distro package maintainers? Downstream derivatives? Porters to obscure platforms? Random developers/potential contributors?
I remember when Mozilla had trouble when they ran out of virtual memory space in the linker (see https://bugzilla.mozilla.org/show_bug.cgi?id=709193 , but this wasn't the first time that happened) and caused serious churn for the Windows build because Visual Studio was their only option. To be sure, this was exacerbated because VS was and is a proprietary product, but if you're going to bet that clang doesn't suffer some similar critical failure in the future, maybe you shouldn't bet every single tier-1 build of your core product.
If your plan for MSVC causing issues is to let it be broken with MSVC –– then MSVC is already not a tier-one platform.
And if you're going to avoid failures caused by issues in MSVC, the existence of other compiler options doesn't change the work you have to do to achieve that.