It's the functional equivalent of someone buying the book and then passing it on to the next person who wants to read it.
With pirating, perhaps one person bought the book originally and then it gets sent to an infinite number of people who permanently have it in their collection. The author gets paid practically nothing.
The short answer is that libraries buy books, pirates don't.
In regards to physical books, a good place to start is understanding "first sale doctrine"[1], which allows both libraries and you to do what you want with a book once you're purchased it.
First sale doctrine does not apply to DRM-encumbered ebooks, so libraries must buy as many licenses as they wish to loan, paying three-to-five times the retail price for each limited-time license, and re-purchasing those licenses when they expire (typically after two years).
[1] https://www.nolo.com/legal-encyclopedia/the-first-sale-doctr...
Here is an interesting answer on the topic.
https://www.quora.com/How-does-a-library-pay-royalties-to-th...
They also provide a net benefit to society by encouraging knowledge and literacy. There is a social expectation that physical goods are sold in most cases without expectation that seller will retain contractual control in order to derive maximum profit. Example nobody liked when Keurig sold coffee pots that wouldn't work with generic or indeed even older official pods via electronic tags in pods.
Also keep in mind that that we all exist in a society there is no reasonable expectation that you have a moral right to be able to use societies apparatus to maximize profit if its at societies expense. Limits are the norm.