Hacker News Reader

Comment by paaloeye | Hacker News Reader

For better or worse, it's like `yt-dlp` — constant race against CF, so it's not reliable. It's probably for better.

yt-dlp works when ever I try it. I do not use it much but ever few weeks when I need to download a streamed youth ringette game, it works great. So while it may be in a constant race, it seems reliable to periodic end users like me.

Depends on the effort from the team. Very good testimony for yt-dlp's one.

yt-dlp works fine for low-volume requests from high-reputation IPs.

But if you try to download in significant volume, their systems know it's automated and probably have signatures specifically for yt-dlp.

I think it's a bit of an escape hatch: they know they can't close the hole (after all, there is always the "analogue hole", or now AI doing browser control); and probably prefer leaving a small monitored and rate-limited door than trying to patch everything.

See: Google changing their sponsored link DOM/CSS/classes matchers every (few weeks - few months) to break ad blocker lists, but they're certainly capable/smart enough to do it daily or hourly, etc.

Honestly, that's exactly how I want it to work. The people doing it en-masse are the ones Google really wants to stop. It would be sad if yt-dlp broke for casual users just because Google wants to stop a small subset of people downloading terabytes of video training data. This feels like the perfect arrangement and yt-dlp can keep working for 99% of users.

They kind of want a small amount of downloading to work because legitimate YouTubers use clips from other videos to make their own fair use reactions and responses.

They likely want to block alt clients that strip adverts while allowing the small amount of downloading to still work.

There is a massive "clipping" universe. Not a golden goose but I'm sure a significant revenue driver (like in Shorts)

i sometimes use it when i give it an LLM to crosscheck yt videos. like give it a link and do a crosscheck especially on history and financial videos

> But if you try to download in significant volume, their systems know it's automated and probably have signatures specifically for yt-dlp.

Well, the big G is not going to sit around and watch it being scraped

Of course not! But my question/point is why they allow a little bit to be scraped at all?

Obviously, the YouTube team is aware of yt-dlp, and with Google's engineering density, a small team can completely block it & win in the cat-n-mouse game against community volunteers that make all their code public.

Probably because of the YouTube clips side, which is a very large community. Allowing them to download and clip videos to then re-upload to the platform could be considered a "loss leader" in my eyes

That is true.

I use yt-dlp to download sporadically some YouTube clips and it works fine.

However, once I attempted to download a complete YouTube list of clips, with a total size of several tens of GB. Towards the end of the list, after a couple of hours or so, the download was interrupted.

Then my IP address was completely banned for YouTube, regardless of the browser used. The ban lasted for a couple of weeks.

Now, used only for infrequently downloading single clips, yt-dlp works again fine.

Use a proper rate limiting scheme, obviously. You're using access patterns associated with human volumes, so you can only ever scrape at human speeds (except that you get the advantage of constantly saturating the pipe and running 24/7)

It's getting much more difficult to run. YouTube often requires PO tokens [1] to download videos now.

[1]: https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide

It really depends on your IP, in my experience. I've had to provide tenured browser cookies for some sites quite a few times already.

Reply on news.ycombinator.com