Why don't websites just remove ad code from pages they don't want ads to appear on, instead of telling the ad server to not display it on such pages by using the referrer information? Seems like a convoluted way to solve the problem.
No, the advertiser doesn’t want their ad to appear on a certain publisher’s web page (e.g. banks often can get in trouble if their ads are associated with certain kinds of content).
Ah ok. But why then doesn't the ad's embed code just contain the URL of the page it's on in the query string / POST data? The (website that shows the ad's) server knows what page it is providing.
I suspect it does? But that's spoofable by the ad aggregator in a way that the Referer header isn't. So much of ad design is (for better or occasionally for worse) defending against bad actors.