I also suspect your wrong in saying that advertisers will completely give up if the number of deliverable boxes shrink. I mean, sure, they will want to pay proportionally less, but I suspect post office revenue from advertisers would decline smoothly with the rise in revenue from the paid opt-outs, assuming you got the price right.
My understanding is that fourth-class "to resident" mail is usually targeted at the most price sensitive customers; people for whom clipping coupons is often a good use of time. Exactly the people who are unlikely to pay money to sort their mailbox.
Most of the higher-end mail, like those goddamn pre-approval notices from AmEx, are lovingly personalized, and ship out 'pre-sorted first class' or something. I'm only talking about opting out of the fourth class "penny saver" type junk.