Next time when Facebook pulls something user-hostile (e.g. monetization with ads, yet another privacy policy change for the worse, ...) some people will simply install Signal. If they use phone numbers as (an) identifier, two people who do this independently can immediately switch to Signal.
If A convinces B to switch, and C convinces D to switch, B and D can now talk to each other, reducing the pressure to keep WhatsApp as more and more of your friends are reachable on Signal. Even if you're using WhatsApp in addition to Signal, with phone numbers as identifiers, you're no longer contributing to the network effect that makes it painful for your friends to switch from WhatsApp to Signal.
Given that network effect is what makes or breaks messengers, phone numbers as the primary identifier are the only reasonable choice.