Yeah, me too. It's boring, but after the last decade I'm more than willing to accept that "cost" for the benefit of having a sense of what's going on with my contacts.
Does anyone know how to efficiently unfollow large numbers of people though? I was way to promiscuous in the early days and want to winnow down to someone I've actually worked/socialized with.