That's exactly the point, denying them access to a payment system the consumers can easily access, hence, limiting the profit motive and the resulting massive human damage.
By your very limited logic, denying drug traffickers access to the banking system hurts poor coca farmers in Bolivia. Well yes, it does, any policy choice involves compromises.