Feynman drank, played bongo, womanised, slept with some of his friends' wives.
But that does not mean everyone who does that is a great Physicist. Neither all great Physicists do these.
Taking an outlier and using him for justification of something seems obtuse.
I love Feynman, and have a picture of him on my wall. But that does not mean I idealize or even like everything he did in his life.
Incidentally, I've not read much about his womanising - perhaps I'm not reading between the lines?
And although the bomb was used on the Japanese, it was meant primarily as a weapom against Hitler.
Feynman was an American Jew. Imagine yourself in the same situation. There's a dictator who hates all Jews and basically any race which is not Aryan enough for him. He wants to kill them all. A gigantic war machine which is fighting the rest of the world alone is also possibly building something similar.
You get the opportunity to work on something that undermines the devil, that will buy you and your country protection against it. Will you not work in it?