Phytoestrogens actually make it harder to grow breasts because they compete for receptor space and then don't do a whole lot.
Similar effect with Estrone (E1) and Estrogen (E2). Many trans people switch from oral HRT to other methods that bypass first pass metabolism (which converts E2 into E1) because their E1 is too high and the desired breast growth doesn't materialize.