plantpark

I thought Pie chart will make the percentage more clearly. Do you have any suggestions for the chart? Thanks!

Sorry for that, I will check it again.

What kind of chart do you think is better for such data? Looking forward to your advice!

Thanks, I will check the source for more details.

It's not about some simple word frequencies of words. Some common words like "like" "need" "second" "part" in the dataset of whole documents isn't so meaningful in a specific sentence. Google "tf-idf" will show you more details about this.

I've seen this article.Great article about distributed crawler. Mine just is a demo version of his.

Great crawler, Thanks for your share.

Thanks for your comment. I've used multiprocessing/threads/geven/asyncio before. And I will have a full test with these libraries or tools. This post is just a quick demo to build a distributed crawler with docker. Asyncio and aiohttp is a great combination for this case , using less memory and faster. But aiohttp only support http proxy, perhaps this is the only case not so perfect.

Thanks for your comment again. welcome to discuss more technical details about it.

You are right. But with more codes or tools , it could do this too. It's just a quick demo for distributed crawler. If you moniter traffic of your target website with js rendered content, you will find json file and json api. And what you need next is just the same code in my article.

Thanks for your comment. If you have the same test with cloud server or some public website , perhaps it will decrease some.

I've used multiprocessing/threads/geven/asyncio before. And I will have a full test with these libraries.

Thanks again!

Thanks for your comment. Great article for networking. I'm not so familiar with NAT of linux, so could you post more details about it with python or docker, performance/advantage or something else?

Thanks again.

Sorry for that incomplete description in my article. Because of GIL of python, python application could not make full use of machines. Of course, some libraries like gevent/asyncio could fix it well. I will have a test to find out which one is the best, include docker.

Thanks for your comment.

Thanks for your comment.

You are right, asyncio and aiohttp is a great combination. I've used them both before. Though aiohttp cost less memory than requests, aiohttp does't support https proxy. This is the only one that isn't so perfect.

Besides , asyncio is something the same as concurrency of celery. I will have a test which one will have a better performance.

Thanks for your comment again.

Thanks for your comment. This article is a tutorial that tell people how to build a distributed system with docker and celry easily.

Scrapy is good framework for crawler. I used it before, but it doesn't have some features I want. Writing a brand new one is easier for me.

Of course , I am open to compare the special performances of these two crawlers. Welcome to have a discuss more technical details here.

Thanks for your comment again.

Sorry for incomplete description in my article.

It's just a tutorial about how to use docker and celery to build a distributed system. I will have more test about the performance of multiprocess/threads/concurrencies or some other library that supports these technics. In this case, if you don't know how to or even don't want to build a test web server, some big sites like example.com could be your choice. BUT please be gentle to these public sites.

Thanks for your comment.

celery has a prefork pool that could take use of multiple processes.

Some library in python ,such as asyncio or gevent could do some work asynchronously and efficently. I will have a test later for these library. In the meanwhile , welcome to post more details about asynchronous of Node.js. Thanks for your comment again!

Thanks for your comment, I will have a test with a local server later and find out what's the upper limit of mine