This is the inverse of the issue. I don't want everyone that has my phone number to be able to see/add me on telegram. That would require those users not to upload their contacts, which is out of my control.
Everyone can't. You can adjust the privacy settings such that you have to add someone to your Telegram contacts before they can discover you are on Telegram. It's easily done in Telegram's privacy settings. This is a primary reason I prefer it to signal despite the screeching from armchair crypto experts on HN anytime Telegram is mentioned.