The problem is that you're never good enough. Considering research is global competition, it's likely that there will be a bunch of people much better than you, in more prestigious institutions. They will be very critical of your work. Some will dismiss your work publicly in conferences or seminars and so on... not my idea of fun :)
> In the end though I do prefer working commercially, in a field where I can get intrinsic satisfaction from solving real-world problems instead of just having my name at the top of a list or being judged worthy by my peers.
It's possible to do both. Some researchers do try to solve real-world problems :)