Ask HN: If you were to host some scrapers and a MySQL db what would you choose?

1 point·saasxyz··5 comments

I need to host some scrapers and also data for each sites stored as JSON along with the scrapers. Let's say the scraper runs only once a day. And I need to host a Mysql db whose data is used to statically render pages. So, the data is queried like 10 times a day.

In such a scenario, how much would you expect the total costs would be and what service would you use?

View on HN

I have used pythonanywhere for this. Look at their pricing.

There are preset and custom plans available. There is a free tier that may work if what you're scraping is on their whitelist; all paid accounts are unrestricted that way.

If you're not doing any computationally intensive processing, and it sounds like you're not, the biggest factor in cost will probably be how large your database is going to be.

It did everything I needed for $5USD/month.

You must be new around here. The HN answer is just use SQLite! ;)

can you elaborate a bit about how would you go about that?

Are you asking specifically about hosting or the code or the architecture etc?

SQLite is an embedded DB in that the contents are all in 1 local file (like an excel file) but you point your code at it and can use it as a sql db.

So I was being kinda tongue in cheek earlier. But for starting out it’s a good choice until you have a ton of scale.

Can I use this sqlite db to post data from my scraper (that is hosted somewhere else)? How much is considered big in a sqlite database? [edit]: I already have 6 MB of data in mysql db

Reply on news.ycombinator.com