Downward volatility hurts you more than upward volatility helps you (on average). That's where the -sigma^2/2 term comes from.