It’s not about the hashes, i.e. the proof of work part, it’s about the fact that bitcoin addresses are public/private key cryptography, like RSA.
Now I think bitcoin actually uses es elliptic curve cryptography (I don’t know, I really don’t care about bitcoin), but the hypothetical was more along the lines of “what if you could break public/private key cryptography”, and less about factorization in specific, anyway.