But in this case, it’s important to remember that phone companies used to regularly leak most of their customer’s phone numbers (and names) in the form of a telephone directory. So a question to consider is: suppose that the white pages were still commonly produced and contained most people’s numbers. How would you then feel about something like this.
Personally I feel like the problem with phone numbers being leaked is mostly the epidemic of spam calls (especially in the US) rather than some particular breach of privacy.
Aside: I think it is good to consider these counterfactuals in general for questions about information privacy, for example how would you feel if everyone’s tax returns were published publicly like they are in Sweden?