Nvm, on Telegram X under Privacy and Security you can "Delete Synced Contacts".