If it's fair enough for Youtube to send me a video full of ads, then it's fair enough for me to run a program that removes the ads. The only way Youtube's business model actually works is either a) people watch the ads voluntarily, or b) they are able to force your computer to not do what you tell it to and instead to show you ads. Because all computers are general purpose computers, option b allows the server to impose arbitrary restrictions on you, not just reasonable ones. You might think watching ads is ok, but future restrictions (like fiat banning you from otherwise public content) might not be. That's the argument.
Of course, if everyone used an ad blocker then Youtube's business model would not work. But businesses are not entitled to a revenue stream.