I'm a normal man and don't think twice about consuming soy, as humans have done for centuries. In 2018-2019, it was estimated that 1.9 million Americans are considered allergic to soy, compared to 8.2 million for shellfish or 6.1 million for milk. https://www.foodallergy.org/sites/default/files/2020-06/FARE...
Phytoestrogens are a fun bogeyman for mostly conservative "macho men". The Harvard School for Public Health has a decent springboard of research to start from: https://www.hsph.harvard.edu/nutritionsource/soy/