Right, because the web activity itself can contain location information independent of the _explicit_ collection of location history.

It comes down to the granularity of privacy concerns and the complexity it introduces.

If you disable the direct collection of location history, should you, the user, be allowed to turn on the collection of other information that can contain location data? That would include all location-enabled service histories, of which search and app history is the most obvious example. Or should Google treat your setting as a prohibition against all possible sources of location data regardless of your other preferences?