Correct (I considered explaining SNI but ultimately didn't bother), and a determined attacker could probably figure out a large number of the sites you visit by the IP as well.

Right, and even still it's pretty trivial to reverse a domain from an IP address in most cases.