You don't own the book itself though, you own a license to the book. The doctrine of first sale does not apply, and they can revoke your license at any time.
They have revoked licenses in the past and deleted books from people's devices. I know you have a physical file that contains the book, but that file is secured with DRM.