My personal impression is that when people of average intelligence can't understand something is because the concepts that make that thing are outside their expertise, not because they are not able to. They are missing the parts, so, understanding the total is impossible.

People of exceptional intelligence is able to learn new things and think through a problems easily, but what makes a real difference is if you already own the building blocks of the new idea or not. A transpiration vs. inspiration kind of thing.

Anyway, I think the argument is that you should be judge by average and impartial persons (a jury or a judge) because, otherwise, you are not in a egalitarian society.

Somebody could argue that an egalitarian society is not efficient, but history teach us where that finish: there is always people on the ready to declare themselves above the others. So we have to aim to some kind of balance.