Second, one of Diamond's premises in this essay seems to be that farming and its consequences are, directly or indirectly, the cause of the malaise in the world (over-crowding, wealth inequality, class structure, war etc).
So I think the point is that with farming you get free time, but also all the bad things mentioned above. What Diamond is arguing is that hunter/gatherers didn't have to deal with most (all?) of the negatives but still had plenty of free time to pursue their leisure. That is, civilization might not have been set back just because we didn't adopt farming. In the end, it was sheer numbers that decided the outcome, not the best method.