Except then your ISP could see what you download, which I doubt many would like.
Are there any ways that could be mitigated?
I don't have a great write-up published (only working code!) but if you know Graphviz, see https://github.com/bazil/plop/blob/main/doc/crypto.dot -- or study Tahoe-LAFS.
Of course, shared secrets scale poorly beyond individual users and groups of friends. But there's fundamental limitations there, you need to do access control by some secret knowledge.