I can't imagine any engineer would just say "Mmh, what happens if both pilots move the commands in different directions? Oh well, let's just take the average". I'm sure there's more to it.
Every branch in mission critical code is tested for all possible permutations. Averaging is simpler from a verification standpoint. Theoretically...
edit: Re: physically-linked controls (à la Boeing), with one pilot pulling back as hard as he can Vs. the other pushing fwd, isn't the result still (approximately) an average of the two inputs? Obviously they'd feel that the controls weren't moving, but that could be attributed to a number of factors (e.g. icing of the control surfaces).
Even if so, a terrible design choice to improve the common case at the expense of the rarer but much more critical case.