Comment by 10165 | Hacker News Reader

Comment by 10165 | Hacker News Reader

Parent Full thread

Assuming nothing listening on 127.0.0.1, this should fix the problem.

   cat << eof >> /etc/hosts
   127.0.0.1 vice-web-statics-cdn.vice.com
   127.0.0.1 fonts.googleapis.com
   127.0.0.1 www.google-analytics.com
   127.0.0.1 d31qbv1cthcecs.cloudfront.net
   127.0.0.1 d5nxst8fruw4z.cloudfront.net
   127.0.0.1 b.scorecardresearch.com
   127.0.0.1 advice-ads-cdn.vice.com
   127.0.0.1 vice-publishers-cdn.vice.com
   127.0.0.1 vice-sundry-assets-cdn.vice.com
   eof

Alternatively, sed script below will give you just the article html, images, and embedded youtube along with all the json data enclosed in <pre></pre> tags.

   curl https://motherboard.vice.com/en_us/article/this-robot-is-livestreaming-all-the-gnarly-stuff-its-seeing-in-the-deep-sea?utm_source=mbtwitter \
   |sed '
   /window.__PREFETCH_DATA/!d;
   s/window.__PREFETCH_DATA = //;
   s/\"[:,]\"/\
   /g;
   s/,\"/\
   /g;
   s/\":/\
   /g;
   s/.[}{]./\
   /g;
   s/[}{]./\
   /g;
   s/[][]//g;
   s/id/<pre>&/;
   s/body/&<\/pre>/;
   s/\\u003C/</g;
   s/\\u003E/>/g;
   s/\\u002F/\//g;
   s/\\"/\"/g;
   s/url/<pre>&/;
   s/embed_code/&<\/pre>/;
   s/autoplay/<pre>&/;
   s/;/<\/pre>/28;
   '

I'm not entirely sure I understand what is happening, I have some grasp of the aim/purpose. Also have pulled pages using CURL before. I'm just curious, does this mean you have to analyze their page (like console) and find the links like the ones you posted above for cloudfront,etc... and then assemble it so you can format it? Not my field but I use ublock/adblock plus and it's not enough all the time, some videos have an overlay which I swear has a double-click counter where you have to open 2 ads before you can actually push the play button to play the video.

It is interesting to grab data and package it yourself through your own reader but I wonder if it loses the site's design/feel... but you're probably just after the information anyway.

"... I use adblock plus and it's not enough all the time..."

Next time this happens it could be useful to make a submission to HN about it.

We might be able to identify and/or solve the problem.

I do not use an ad blocker nor do I use python or youtube-dl yet I never see any ads and I download all videos before watching them. Indeed I access the web for the information not the inconsistent design/feel.

What I posted above illustrates examples of two alternative approaches:

1. Block sources of undesired resources: ads, tracking, etc. The "links" listed are domains used for undesired resources.

2. Only make requests to sources of desired resources: the article and its accompanying images and video. The script extracts only what we want from the html page.

I use approach #2 more than #1.

As far as I know ad blockers use blocking, #1, exclusively. They need to maintain a list of domains to block.

I wasn't sure if you were serious about posting to HN about adblock haha, I complain enough as it is about my pathetic life.

In the case of the videos, they're sourced from other domains, it's funny the site's argument is "Google does this too technically so why are we different?" But I think the embedded iFrame's that contain the video players have their own ads. I don't know, it's hard to read JS code when it's minfied (un-minify it).

You mentioned python?

Anyway thanks for the response.

Something like that sed script would be nice to pipe all of my browsing through, before it ever hits Chrome.

If you are serious and you want a script like this for any given page that you want de-crufted and reformatted, let me know. This applies to any HN reader.

I make these for myself all the time.

I do one-offs in sed and if it is something I will reuse I redo it in C using flex.

If you want something generalized to all web pages that removes google analytics, doubleclick, scorecardresearch, etc. of course that is possible too. I am happy to make this for anyone who wants it.

Another solution is to run a proxy on your computer that filters out garbage. Such proxies used to be more popular when advertising began to take over the web, but most of these projects seem to have been abandoned and forgotten (junkbusters, etc.).

Nowadays it seems like people try to accomplish this via Javascript running in the browser.

Turning off Javascript is also highly effective at disabling advertising. But note this may not stop things like img src tags pointing at tracking pixels, malicious iframes, and other elements that graphical browsers will load automatically.

Also, tcp clients do not normally store or send cookies. And even if they try to provide this functionality, e.g., curl or wget, unlike a graphical web browser, cookies are off by default. Consider the case where the user retrieves the page with a tcp client and then views it in a browser.

>Nowadays it seems like people try to accomplish this via Javascript running in the browser.

I filter out lots of cruft by doing the opposite - using NoScript to block extraneous junk.

λmegous | https://news.ycombinator.com/item?id=13226170

For each use case that is not a free browsing I create an electron app, that never executes any code from the web or uses any external style. It only uses XHR to fetch html pages/json data/other static stuff and then transforms that data and uses it in the custom UI designed for the use case.

> Another solution is to run a proxy on your computer that filters out garbage.

That's what I was thinking; a proxy server that can run on a pi. Plug it into the wall, configure wifi, connect to it, and bam - instant safety. (Assuming you trust the code.)

Reply on news.ycombinator.com