Show HN: Top PDFs Posted to Hacker News in 2018

One of them was a broken link (at least from here) :

#15 Cognitive Distortions of People Who Get Stuff Done (2012) [pdf]

Correct link seems to be http://cliffc.org/blog/wp-content/uploads/2018/05/2018_AWarO...

I too built an application for HN: https://hnprofile.com/

If you're interested, I'd be happy to discuss it as what I focused on was ranking content (not indexing exactly). Might be some interesting synergy. They system requires a fraction of the data of a regular search engine and is often more effective.

With Twitter Music shutdown, I was looking back on their acquihire of WeAreHunted, a music ranking service which at its core was a crawler that indexed torrents/tumblr/soundcloud to find what's up and coming. As I was pondering this, I was thinking about how they would normalise the data.

My main question was, how much difficulty do you encounter indexing a social site? I can imagine Tumblr, Facebook, and other sites have a plurality of new content appearing at arbitrary intervals (posts, comments, etc) - and I don't imagine that there are RSS feeds to diff here. So how would it function?

Edit: Nvm, just saw the git icon. Awesome!

You now gave me (and others) a reason to download your app. Ive downloaded the pdfs and am tracking my reading progress with your app.

I would consider making pdf reading lists and sharing them (with a great pitch to use your app) as a marketing effort in multiple verticals.

    from __future__ import print_function

    import re
    import requests

    from bs4 import BeautifulSoup

    def download_file(download_url, name):
        r = requests.get(download_url, stream = True)

        with open("repo" + name, 'wb') as f:
            for chunk in r.iter_content(chunk_size = 1024*1024):
                if chunk:
                    f.write(chunk)

    html = requests.get("https://getpolarized.io/2019/01/08/top-pdfs-of-2018-hackernews.html")
    soup = BeautifulSoup(html.content, features='html.parser')
    sAll = soup.findAll("a")

    for href in sAll:
        if href.has_attr('href'):
            link = href['href']
            if link.find(".pdf") > 0:
                print(link)

                last_index = link.rindex("/")
                name = link[last_index + 1:]
                print(name)

                try:
                    download_file(link, name)
                except:
                    print("error downloading " + link )

That's the power of "Above the fold" in action.