Perhaps a way to balance this is making flagging (up or down) cost karma points, but that would lead to them (KPs) being visible to give immediate feedback on the user's actions. Likewise, upvotes give back the karma you lost modding some comment. A different cost for up/down seems also a good idea. This, simply, could create an incentive to post even when one has nothing to say and prevent constructive modding by people who post little.
Hmm.... Perhaps an agreeing vote would redeem part of the karma paid to up/down vote.
This balancing act is a very hard problem.