I've heard this argument a lot but it's hard for me to agree.
Saying that ad arbitrage doesn't "add value" is kind of like arguing that someone with a media-buying/sales role doesn't "add value" because they're just sitting in the middle taking a paycheck/commission.
Partnering with ad arbitragers isn't much different than outsourcing internal media buying/sales roles to an agency to increase reach or fill-rate.
Your point about ads being annoying to end-users isn't unique to digital. Advertisements on all mediums have always annoyed end-users. Certain implementations like full-page takeovers & loading 100 ads on a page are the most annoying. Generally it's the publisher's fault for the bad UX.