Curious: how would you imagine it working if there were such a graph db?
541 karma · joined October 3, 2014
Curious: how would you imagine it working if there were such a graph db?
Nevertheless, Citus is now a promising open source alternative.
It saddens me that the default response is ban, increasingly so for services that need account, but even just reading a webpage can become tedious or impossible :(
Please, do not ruin this option, because even though you may not use it today, you may need it in the future.
Note: I also use tor for low volume crawling. When high volume is needed, it is more favorable to subscribe for domestic VPN proxies, so you may be blocking tor, but you won't block those ranges, and the robots will get their content anyway.
(1) Limit your load / parallelism: do not use more than 4 threads, and if the site takes more than a few seconds to respond, use fewer.
(2) Limit aggregate load: after each request, sleep/wait at least the amount that the previous request took to get served.
(3) If you need more than that, ask the site owners for direct channel.
This way if multiple crawler happen to crawl the same resource-limited site, the site may have a fighting chance.
I happen to speculate this because I may have used tor for web crawling, which may have caused a specific site to crash before I cut back the crawl rate. Punishing tor for such incident is mostly FUD.
As a long-time user (and writer) of static site generators, I welcome these ideas, as I think these would allow us to organize static sites into some kind of decentralized social networks. Think of this as a bulkier version of RSS, because - for most practical cases - it is relative cheap to just fetch everything from a source and present it in a consistent way.
Even if people were using centralized git repositories, a separate naming service could allow easy (and cheap/free) redirects, so data migration wouldn't be an issue.
Can this be done with mastadon? Of course. But it requires a database server and other moving parts that need occasional maintenance. Treating the social network as a network of static sites makes operations (and federation) so much easier.
Would this be this specific project that wins us over? Maybe not, we shall see. But I think the concept could evolve into something useful.
There are people who hate commute, and at the same time do care, and "invest themselves beyond what it necessary". It just happens that they don't want to invest themselves in their commute.
Depends on the team and the product. My personal approach is to have 2-3 larger things to work on, so while I wait for reviews on one, I can switch and work on the other. This usually means minimum 1-2 weeks of planned work, sometimes even more, without being blocked on reviews. If everything is blocked, then it is time for some code health cleanup, refactoring and fixing those TODOs that are just lingering around, and also nudging the reviewers...