And the converse: avoid spending time with slackers.
I think being around people that aren't necessarily "productive" is perfectly fine. It's being around people who do NOTHING that is bad but I find that if you are the kind of person who does things then you will find yourself around inactive people less and less over time.
Couch potatoes seem to attract one another and repel everyone else.
It even takes a certain kind of slacking, in a sense, to read lots of random Wikipedia articles, keep up with current events, browse arXiv papers not directly on the topic of your current work, keep up with recent music, etc. Arguably reading a lot is not really slacking. But maybe keeping up with recent musical trends is? Seems all quite subjective, unless you literally are doing nothing.
I tend to see it more as different thought/work styles along several axes. For example, on a question of dabbling in side projects, for some people it'd be good advice to suggest laying off and concentrating more on one thing, while for others, encouraging a bunch of side projects is exactly the right advice. Some people need to do less random-walk reading, while maybe others need to do more...
I think we measure ourselves against our peer group and the more slackers we hang around the more mediocrity seems like success.