David Hilbert wanted to redo mathematics from square one, basing everything on set theory, axioms, and rigorous proofs. In 1900, Hilbert gave his famous address where he stated the Hilbert Problems. One of these was to prove that mathematics is "complete" in the sense that every possible statement is either provable or disprovable from the axioms.
In an effort to solve this problem, Alan Turing wrote his famous paper "On Computable Numbers" where he conceived for the first time his "Turing Machines". Although his approach didn't work to solve the Hilbert problem, it greatly inspired Von Nueman who at the time was pretty much working on pure mathematics.
Godel came along and proved that mathematics is not "complete", because there are some statements about natural numbers which cannot be proved or disproved from the axioms, but rather are undecidable.
Von Nueman was so upset by Godel's result that he decided he no longer wanted to work in pure mathematics, because if it's not complete then whats the point anyway. Instead he decided to work on applied math and physics. Since he loved the work of Turing, he wanted to build a physical Turing Machine to help out with computations, and he figured there would probably be a lot of other uses for such a machine that no one had thought of yet, and that would only become apparent once the machine was built.
While at the Institute for Advanced Study, Von Nueman fought a hard political fight to secure funding and permission to build the first computers, the MANIAC and the ENIAC, to the horror of the faculty at the time, including Einstein.
Since I am posting this at 4:30am in a thread that is 12 hours old, I will be happy if someone reads it :P
[1] http://books.google.com/books/about/Turing_s_Cathedral.html?...