I think what bothers people is the idea of having to work to pay for somebody else's leisure. Obviously lots of arguments to be had about the specifics of that, but I think it's more about this than a moralistic concern about "unearned" leisure. Nobody minds much if somebody wins the lottery and lives a frugal and leisurely life on the proceeds.