A lot of people do this. Some people do refuse to work on military technologies. Even Google declared it wouldn't produce anything for the military with the robotics startups it's acquired. Some of the scientists who worked on the Manhattan project later regretted it.
If you really do believe the thing you are working on will have a net negative effect on the world than maybe you shouldn't work on it.