I wouldn't be surprised if he's at least partially an inspiration for the Dr. Felix Hoenikker character in Vonnegut's Cat's Cradle.
In my opinion, Hoenikker is just too amoral and aloof to really match any one of the Manhattan Project scientists. Teller (who suggested using atomic weapons to excavate harbours) and Fermi (who before the Trinity test took bets on whether the bomb would light the atmosphere on fire) are maybe the closest contenders.
[1] https://gizmodo.com/the-real-life-scientist-who-inspired-kur...
Von Neumann was himself pretty pro atomic bomb, though, and wanted to preemptively bomb the Soviet Union.