I wouldn't worry about it much - there are a lot of us out there who aren't exactly "single-issue voters", but for whom a single issue can make a candidate completely unpalatable.

For me, that's guns. For you, that's drugs. For many others, it's abortion.