It might be that it notifies those other users of your phone number when they might not know it. I.e. if you store a number of someone you don't want to speak to so you can decline their calls.
Supposing you did this for extra piece of mind, even though they don't have your number, it would suck if they end result would be you giving them your number.
I don't think there's any notification that someone has added you to their contact list. That is, if you have someone's phone number in your contact list, and they sign up for amazon calling, you see their name in a list. However, if they don't have your number already, they can't see you as a contact. The only information leaking is that someone can learn that you're still using the phone number they have for you and that you're using amazon calling. Contacts in amazon calling don't appear to be two way.
Thanks. I was struggling to figure out how this was worse than normal cell calls - as the ability to block numbers is still mind-boggling rare.