So you could work for an investment bank, make a lot of money, use as little of it as possible and save the rest, then after a few years, quit and use the savings to fund a start-up? Sounds like a plan to me...
The head of BizDev at the financial software startup I used to work at was one of the lucky ones who got out (and he told me the above observation). You occasionally see other ex-financiers in the startup world, eg. Jeff Bezos or Joshua Schacter, but in general you don't see very many ex-investment-bankers. Most of them seem to still be working on Wall St.
BTW, this applies to other professions as well. An engineer frequently will see "just one more technical challenge that needs solving before I can get to the good, world-changing stuff." (I'm dealing with that now - I keep getting distracted from the ideas I really want to do by the various low-hanging fruit.) An experienced teacher will think "Just one more year with the kids, then I'll retire." Startups are nice in that they have clearly delineated endgames; most salaried professions are not like that.
There are plenty of people doing it, at any start-up event I go to (in London) I run into current and former banking peeps.