Oh, it is absolutely a derogatory term - it's just that the people it is applied to are really not 'people that care about people'.