There's actually a great deal of effort towards improving quality and dealing with ad blocking at a high level in the industry. One recent effort this year is https://www.betterads.org/ but there are other initiatives lead by new adtech companies and other organizations.
As a result, I'm confident that ad quality will increase over the next few years overall. But like you say, its an intractable problem. Its not as simple as improving quality on the already reasonably good 75% of adverising that isn't so bad.
The bottom of the barrel inventory is likely not going to rise with the change of the tide, for example. If you're trying to stream pirated TV shows or watch porn you're dealing with the North Koreas of the adversiting industry--they're already cut off and live in their own world, and won't follow industry initiatives that publishers like NYT will want. Unfortunately many, many people have no problem with advertising generally but install ad blockers in order to access these sites, and the sites no one had an issue with are just collateral damage.
Moreover, many people install ad blockers because, lets be honest: why not? No more pre-roll video. Faster page loads. More premium consumption experience (most) everywhere you go. How do you address those users with better ads?
And the cat is already out of the bag anyway. 25% of people on the internet use ad blockers. If you improve ad quality, how do you get these people back? I think we're going to see a radical, unequal shift in the next fews years around the perceived value of content because of the growth of ad blocking. If you told me 10 years ago that everyone was going to be paying $10/mo for music, I wouldn't have believed you, yet here we are. I think if you ask me this time, I'd say in 10 years people will be more inclined to pay for digital content than they've ever been, because as ad blocking grows, the "anti-ad blockers" are going to grow too. More messaging to users about ad blocking, more paywalls, more access control requiring equitable value exchange across the internet. Eventually the middle will get squeezed out by the high cost of ad blocking. Users will choose to whitelist or subscribe for premium content, and the lowest cost publishers who steal, copy, or lie to generate clickbait are the only people who can survive when they've lost half their inventory.
> It seems like an intractable problem. I'm not convinced technology (machine learning! the solution to all problems!) is capable of providing the solution.
You're absolutely right. This is not going to be solved by machine learning. Ad blocking is an economic and political problem, and will only go away when publishers, advertisers, and users are able to come back into equilibrium. Publishers need more leverage to fight for user experience against advertisers' demands, and users need to understand the value exchange that occurs when they get targeted apple adaptor ads while they read articles about the latest iPhone. Publishers need to offer users more diverse but more firm and equitable options that make quality content sustainable and appealing to even the most privacy oriented audiences. Eyeo need to realize that profiteering on a protest movement is not a ethical business model. Easylist maintainers need to realize that they are not profits sent from god to punish publishers for protecting the value of their content.