What you can't do is to reward people voting on a poll that the majority does not want to vote in.
I don't see why that would be a problem. In fact, we already do that all the time. Look at the voting turnout for a large number of elections. Are you seriously suggesting that >50% of voters have to vote on something (that they may or may not be properly informed about) to make it legitimate?
I generally don't vote on my local school and education laws, because I don't have kids, and don't follow that system enough to be properly informed. My city has a low percentage of parents, so I'm assuming that a lot of other people don't either. If the choice is between everyone voting on stuff that they don't understand, or only those who care enough vote, I'll take the later every time.