You can, just deny the request and add contacts in Telegram. At least in android, I denied contact list and manually adding relevant ones to telegram.