Just upvote stuff you like, and ignore stuff you don't like. If other people find similar stuff interesting, the front page will be filled with them.
If something made front-page, there are enough people upvoting it that your downvote will not kick it off the first page anyway.