There might be a really strong correlation, but the deviations from that correlation might not be totally random.
That is, there might be people who consistently test above or below their skill level. This creates some weird imbalances that are far from fair.
It would be a lot more acceptable if deviation of test scores from skill were to be totally random for every test.