Hence whoever maintains this list will be able to observe the traffic on a third party website.
Auditors and CT gossiping are responsible for ensuring that the log servers are not misbehaving.
I suppose a client could act more like a monitor and download chunks of the log at a time, thereby hiding which site in the chunk they were interested in. That wouldn't be hugely efficient though.