Hacker News Reader

Cool

One issue I have w/ webdriving headless browser in general is host RAM usage per browser/chromium/puppeteer instance (e.g. ~600-900mb) for a single browser/context/page.

Could crawlee make it easier to run more browser contexts with less ram usage?

e.g. concurrently running multiple of these (pages requiring js execution): https://crawlee.dev/docs/examples/forms

View on HN

In crawlee, you can use the useIncognitoPages option to create a separate context for each page https://crawlee.dev/api/browser-pool/class/LaunchContext#use... Not sure if it will be enough to offset your RAM requirements.

From our experience, RAM is not the limiting factor. It's the CPU. You need at least 1 CPU core for the modern browsers to work reliably at scale so if you're using a container that has 1GB ram and 0.25 core, it's just not worth it. If you have access to containers that have strong CPUs and not a lot of RAM, then it's a different story.

That being said, for scraping purposes, you can almost always build the scraper with HTTP requests only. Sometimes it might be hard but theoretically it is always possible (it is what the browser itself does right).

Reply on news.ycombinator.com