Yes, someone could shoulder surf it, but we don't tell you that your bank account has no security because someone could shoulder surf you entering your password.
I'm aware of a story where a local credit union assigned account numbers strictly sequentially. A customer setting up direct-withdrawal typo'd their account number by omitting a digit, i.e. their acccount number was '12345' and they entered '1234'.
Since the numbers are sequential, '1234' happened to exist. For about a year, the company in question cheerfully direct-withdrew from the inappropriate account, and the original owner of '1234' never noticed, never complained, or had their complaints ignored. To my knowledge, the error was never rectified.
End of the day, direct-draft is a badly-architected system from a security standpoint.
I mean after all, we assume that's true for passwords.
URL-shortened links can be an issue, but raw google docs links have crypto-length randomy numbers.
The first time you email that link out, you have technologically released your ability to predict who will view the document (was the e-mail sent over secure channels end-to-end? Did it go to a trusted party who won't reshare it? Did you typo the e-mail address and send it to an undesired party? Did a recipient print it out and leave a printed copy lying around in an accessible conference room? Was the printout shredded after use? Was the shredding functionally irreversible? Is the shredding being done by a third-party that might lose some loads of documents on the way to the shred facility? Did you leave the link in your local machine's pastebuffer then walk away without locking your terminal? Etc., etc., etc.).
Even when the link is functionally unguessable, allowing non-authenticated access is just "security through obscurity."
If I shared a document with only you (directly, not using the "anyone with the link" permission), I couldn't stop you from taking a screenshot and sending that to whomever you pleased (or printing them out and leaving them in a conference room, like you suggested). Sharing documents online in general can be troubling if you don't trust the people you send them to. It's easier to share a link than share a screenshot, though, and a link would provide continued access going forward, and could remove deniability that the screenshot was faked.
Whole-document transformation and copying are a concern, but I think that's usually treated as a different category of issue from "The server I trust to store the data securely gave it up to some anonymous person who passed a correctly-formatted request to it because the server can't know any better."
I was just making the case that "Anyone with the link" still provides some security. Just like you don't know if the people accessing the document are the ones that you shared the link with, you don't know that the people you shared the document contents themselves with aren't showing others without your knowledge.
I think that would be implied when the setting says anyone with the link.
It would be the same if I attached the document to a email, someone could still forward the email.