Lets take an extreme case where you have a million people each with a 55% chance of getting the right answer, and one expert with an 80% chance of getting the right answer.
If you have a normal vote, the right answer is EXTREMELY likely to win. But if everyone delegates to the expert (after all 80% is better than 55%) then the right answer only has an 80% chance of winning. Worse outcome.