Seems like it works better. I agree that the probability of you buying a jacket given that you visited the website and didn't buy is lower than the probability of you buying a jacket given that you visited the website (sorry that's a mouthful). But the probability of you buying the jacket given that you visited the website and didn't buy is higher than the probability of you buying the jacket given that you didn't visit the website.
TL;DR: P(buying|visited the website and didn't buy) > P(buying)
my point is more along the lines of, i'd like to see ads serve me a little better rather than serve the advertiser. the priorities are advertiser > site > buyer. should be opposite.
Yes, with retargeting you hit a lot of people that aren't interested in the product, but you also hit a lot of people that are lukewarm to warm leads. We're talking about the difference between, say, .05% CTR running ads in a content category versus .1%+ with retargeting across ad exchanges.
Specifics vary between advertisers: try visiting Shopify and you're going to see their ads 10x+ per day every day for a month... but less aggressive retargeting frequency (1-3x/day) is used all the time.