Ask HN: How to locate ads on websites programmatically?

2 points·InGodsName··6 comments

Let's say if i want to screenshot the ad banners which appear on the website. How can i do this?

The challenge is the determine what is a banner programmatically.

What technology can be used for this?

What are you using to fetch the pages? Something like CasperJS/PhantomJS? If the banners are images or iframes from an external domain you should be able to traverse the DOM and locate them this way. You would have to have a list of ad providers domains to look for and you might miss some.

Using chrome headless

Modify an ad-blocker to instead of blocking highlight ads?

Nice idea! Is there any resource on how modern ad blockers detect and block ad banners?

Some ad blockers, like uBlock Origin, are open source, so you can examine the code and see what they're doing:

https://github.com/gorhill/uBlock

The basic algorithm is to match URLs of loadable resources (images, JavaScript files, iframes, etc.) against huge lists of ad/malware/tracking sites. If the URL matches, the resource isn't loaded. The lists that uBlock Origin uses are enumerated here:

https://github.com/gorhill/uBlock/blob/master/README.md

As greenyoda suggested, I'd start with source and documentation of uBlock. From what I know, the major blockers share the format for their blocklists. There's two types of blocks: visual blocks, which contain a CSS selector to hide, and network blocks. Since the latter commonly block scripts that load ad elements, you'd probably need to load the page twice, to observe what elements are not created when the ad-scripts are blocked.

Reply on news.ycombinator.com