As you mention, it's easily determinable, and it has the fortune of being an existing ID number system already connected to everyone.
Allowing everyone the ability to tell people apart based on a number is infinitely better than matching names.
In my country we use our ID numbers for everything, dentist appointments, vehicle inspections.. if you call any business at all it will likely be one of the first things you're asked for.
The number should be an identifier for a person, not a security measure. If you just accepted the fact that SSNs are insecure and that by knowing someone's SSN does not necessarily mean you are that person, then the US could use them more effectively, cut down on a significant number of errors in every industry, and actually reduce identity theft.