Hacker News Reader

Comment by tedmiston | Hacker News Reader

Pretty cool project. It looks more enjoyable to use than BeautifulSoup.

How does it approach throttling or rate limiting? I didn't see this mentioned in the readme examples. Would be nice if there were some simple config to kick requests back into a queue to be re-run once limits aren't exhausted.

Minimal support for caching / ETag / etc would be a nice addition.

View on HN

The throtting can be set directly from untwisted reactor(planning to implement soon once i get untwisted on py3). I think the support for caching is really good too, i plan to implement it this week.

Awesome. It looks like you're reusing your own dependencies which is cool. Can you explain how untwisted relates to twisted a little more? I read the repo readme, but not sure I'm following.

Untwised is meant to solve all problems twisted solves but it does it in quite a different way. They are two different tools that would solve the same problems using different approaches. Untwisted doesnt share code nor architecture with twisted. In untwisted, sockets are abstracted as event machines, they are sort of "super sockets" that can dispatch events. You map handles to Spin instances, these handles are mapped upon events, when these events occurs then your handles get called. The handles can spawn events inside the Spin instances, in this way you can better abstract all kind of internet protocols consequently achieving a better level of modularity and extensibility. That is one of the reasons that sukhoi's code is sort of short, it is due to the underlying framework in which it was written on.

I'm not able to figure out dependencies.. is this pure python ? Or are you using one of gevent, libev, uvloop, etc.

Since it is py2, i suppose asyncio is out of the picture

untwisted is pure python. it uses either select/epoll for scaling sockets.

This is very interesting. did you consider using libev/uvloop - which are generally consider battle tested async frameworks ?

Is there anything missing that prompted you to reimplement ?

It seems a good thing to do, indeed. i'll consider that.

> It looks more enjoyable to use than BeautifulSoup.

I don't believe BS is a full scraping solution, it's only the HTML parsing/querying isn't it? In that case, this project actually uses lxml for that part - a relatively well known alternative to BS.

I highly recommend lxml, the API isn't perfect, but in my experience it's much more powerful than BS, and significantly faster as well. We run custom scrapers for a large number of websites, and apart from a few where we use JSON feeds, the majority use lxml, it has been very useful.

Yes, BeautifulSoup is focused on parsing not crawling (BS supports the lxml parser out of the box). Scrapy is more of an opinionated scraping framework whereas BS is a parsing library for scraping. I think the choice depends on what exactly you're trying to build and scale. I like both personally, though I'd use BS for simple MVPs and Scrapy if I wanted to crawl thousands of pages.

Reply on news.ycombinator.com