The spec suggests that it would more be to avoid selecting things that are not part of the text flow of the document, which would be a usability win. i.e. captions that are in the middle of the flow, etc.
But you're right to point out it could be used by the unscrupulous sort of people who also like to disable right click, and that would indeed by highly annoying.