I've been in a fight over a public photo. So, if you don't like fights, perhaps get a telephoto lens. (Saw accident, took picture, man that caused accident decided he was Bruce Lee. Turned out he wasn't.)
I returned a 70-200 after trying to lug it around for 4 hours. It's just too tiring and too silly for general use, and it's certainly not inconspicuous if you're going to be in public.
I have a 24-105 f/4L that I use for day time and a 35 f/1.4L that I like to use at night. I do own a few other lenses but the two mentioned make up 99% of my usage.
I would never advise the use of a zoom or telephoto on the street.