Hacker News Reader

Comment by core-questions | Hacker News Reader

SEO companies are big customers of products like this. Scraping Google at scale requires a huge budget for proxies and IP blocks, and I believe they apply ML to detect people abusing their service.

View on HN

I actually don't think I would need to make a ridiculous amount of requests. The project is a pinning service where people submit a video URL so they can do things like manage lists of the videos they watch (think goodreads for web videos). I need to scrape the video description and tags for each submitted URL. It would be nice to just do it from the user's browser, but the same origin policy won't allow that. A browser extension is the next obvious route, but that has its own issues. I was thinking to first attempt the request via CloudFlare workers, then fall back to a residential proxy service. No idea how usable workers would end up being for this sort of scraping.

This sounds like your would be better of using Youtube's API https://developers.google.com/youtube/v3/getting-started

Yes, I was going to mention this. I worked for a small company that had a YouTube product that retrieved video descriptions, duration, thumbnail, etc, using the API.

We started with 100k daily API calls and that got later increased to 1 million after requesting it.

Looks like the limit is 10k/day now. What did the process look like to get an increase?

automated form for me.

Nice. And that got you from 10k to 1M?

I am at 10m+, keep in mind it isn't 1:1 with requests. Things like GET read only video metadata is one request, but if you are uploading a video, it might count as more out of your quota.

You are expected to be well behaved though. You can also lose your quota or API access for severe abuse, like using API to mass upload spam.

Very nice. I only need metadata. And if I'm not mistaken I can even batch requests to multiple video ids if I ever reach that level of scale.

Yeah it would probably be the easiest way to get started. Their ToS[0] just seems so convoluted it's hard to know if you're following it or not, so I don't want to tie myself to it with no recourse.

[0]: https://developers.google.com/youtube/terms/api-services-ter...

I can confirm that, scraping Google instantly needs huge effort and money. In our best we can scrape 2500 SERP per IP.

But i must say using proxy services and other things did not helped us much. Because most of them were banned before we use.

How does it wotk out for IPv6?

Complete /64's get blocked.

IPv6 is just not widely used, so when you do use it, you stick out like a sore thumb. Think like a bayesian: for Google, it's easy to just block whole /32s of IPv6 space.

At what point would you consider IPv6 "widely used"? It's currently 30% of traffic:

https://www.google.com/intl/en/ipv6/statistics.html

I bet you half of that is scrapers!

Yeah, and then you recycle the IP back into the pool for the next guy to work with. An operation I know of was getting 6+ million SERPs a day, budget for proxies was hundreds of thousands a year.

Reply on news.ycombinator.com