You can argue about the size of this group, but the counter argument would be that there is a valid fear that if it is normalized and allowed to continue, this group will grow larger and larger over time. You can argue by trying to compare this to libraries, but libraries still buy their copies of the book, and checking a book out from a library comes with restrictions that don't exist here. There really isn't a way to spin this where it doesn't result in some amount of lost royalties for the author.
There's plenty of totally reasonable hypotheses for why giving away content can be profitable OR unprofitable, and plenty of real-life examples of both as well. The fact is, with regards to this situation, we don't know, and this conversation shouldn't center around a problem that some people have only hypothesized exists.