I think I'm sympathetic to that idea, but the ethical consideration there isn't that it violates copyright. It's that it's disruptive to society/the economy.

Or maybe one could argue that because it's owned?

If the artist stole the artbook from a shop, the problem is not that they are using it as a reference; the problem is that they stole it. Likewise if they download the pdf from a pirate site. It is a separate, different problem. You can tell because there would still be a problem even if they didn't then go on to produce art, or read it at all.

If, however, the rights holder chooses to just give them away; or, y'know, puts them on their own website for anyone to look at - there is then no license fee to be paid for looking.

Note that this still does not mean someone can make copies and sell them. That's a separate right. But using such materials as a reference is just fine, and people do this all the time.