If the number N is considered unlucky to the point that it is skipped, then shouldn't people be worried that they are living someplace that should have the number N, even if it was labeled N+1?
In this case it seems the bad luck is attributed to how the number four is spoken. So if you avoid saying it you avoid the bad luck. Looks like this superstition works differently from how 13 works where the count itself is unlucky.
Something tells me that people who care about these superstitions don't think quite that far.