I have the President of the United States' address. 1600 Pennsylvania Avenue. Oh no! I done dox'd him!
Exposing information about someone that is largely already public is somewhat bad, but it's not "really, really bad". And it's certainly not worse than exposing the last 4 of their SSN.
There are many ways to dox people. If you have a specific target, you probably know their name; if not, you can phish it, and any other information you want. If you have their IP, you know their ISP. With their name and their ISP, and maybe some extra info gleaned from various sources like social media, you can get pretty much anything you want. Account access, phone numbers, billing information, socials, etc. With their phone number you can take over their phone, and then all their SMS-linked accounts.
Is this scary? Yes. Did I need to slowly extract their home address using a vendor's web form and their IP? No.
The last 4 of the social is much worse. It makes all of this incredibly easy and gives access to much more sensitive information, like medical records, payroll, government service information, etc.