Interesting. It seems like one potential hazard is that bonafide port conflicts are not detected. If SO_REUSEPORT is preferred for performance reasons, and most/all servers are using it, then starting up a server that uses the same port as an existing service becomes a silent error.
It could even work as expected for a while (since the kernel gets to arbitrarily decide what port to deliver incoming requests to) only to intermittently fail later.