How many big success stories involve people who weren't really working all that hard? Do you really believe that's a common scenario?
Even if they weren't working on it at the same time, as long as somebody's looking when you gain moderate success, you've got a competitor who can use your current work as R&D to catch up — and then they can outpace you if you're giving 30% fueled by pocket change and they're giving 100% with the resources of a millionaire. This might not mean death, but it does mean that falling backward into monumental success is extremely improbable. That's why people say lifestyle businesses are unlikely to yield huge rewards.