Variance is a funny thing. For example, a stock that went up every day by 1%, would have higher variance, and thus more "risky" than a stock that went 0.1% up every odd day and 0.1% down every even day.
Under some suitably broad efficient market assumptions, the expected return for all stocks is basically the same (after adjusting for risk). So variance automatically means going up and down compared to that expected return.