It wouldn't be a strong signature, it would simply be the other half of the number pair. Numbers A and B both have easy access to A:B and B:A.
The hashes for a given user could still be attacked using their phone number, but a global table wouldn't work.