...at the expense of people who are moderately good at both.
I think it is fair and better than purely weighted grades because it allows for momentary failure while still requiring knowledge of the material.
That means your grade will be worse than it would've been otherwise when compared to your peers (which is the only useful meaning of a grade).
So what's the problem?