There is an existing solution to give you this protection, it's called port knocking.