If "ballsy" is a positive description and you literally don't have balls, then it can be seen as obliquely ruling you out from ever really behaving in that way.
Whether it's something you'd say in a business setting is a different conversation entirely.
I was trying to find an example of a part of anatomy being used that only applies to women, but could only come up with the negative example of "hysterical".