Isn't this two-factor login? Something you know (username) and something you have (phone)? There's also a good chance that the scammers had a set of leaked security questions on hand. I'm sure some banks even use SSN as an authentication factor.
SSN is alsp a stupidly bad usage as an authentication factor. A lot of people have access to it, it's not unique to the service and you can't just change it whenever you want.