Caveat: I immediately block anyone who adds me to a list (it's only happened 3 or 4 times and it seems spammy). And I've seen a number of other accounts with "don't add me to lists" or equivalent in their bio.
I'm a relatively new Twitter user and I've been making heavy use of the list functionality. So what's the down side?
In fact, I don't even look at the main feed. I only look at my lists.