Hacker News Reader

Comment by 1vuio0pswjnm7 | Hacker News Reader

One has to execute functions from base.js^1 to modify the "n" URL parameter and the "sig" parameter to get the fastest download speeds. (One can still download videos with the original n parameter, or without the n parameter, but download speeds will be slower.)

By itself, using the ANDROID API instead of the WEB API^2 does nothing to affect download speeds. I can block yt-dlp's POST request indicating the client and API name and this has no effect on download speed.

A website that forces users to run Javascript in order to get faster download speeds. This is not a new idea.

1. https://www.youtube.com/s/player/{player_version}/player_ias...

2. From the YouTube video page JSON:

ANDROID API KEY

AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w

WEB API KEY

AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8

View on HN

yt-dl and yt-dlp use Python to interpet the Javascript functions that modify "n" and "sig", but a faster language could be used. For example, V8 is written in C++.

One of the reasons yt-dl and yt-dlp are so slow is because they do (too) many do other things besides running the Javascript, modifying n and sig, and spitting out a fast download URL. Before they can run the JS, the video page needs to be retrieved, but waiting for Python to start up and do this is slow. A YouTube video page can be retrieved much faster using netcat, outside of Python. YouTube video can be downloaded much faster using an HTTP client like tnftp, directly, outside of Python. YouTube video can be converted much faster using ffmpeg, directly, outside of Python. And so on. These programs start instantaniously when compared to the slow start up time of Python. The Python startup latency is unbearable.

At the very least yt-dl and yt-dlp should accept an already downloaded YouTube video page as input instead of forcing the user to use Python (or Python calling another program) to download the page. The recalculation of "n" and "sig" do not have to occur within seconds of retrieving the video page. There is no need to require the user to use Python for downloading webpages. The values in the YouTube video page are good for a substantial amount of time.

Both scripts have an option to output the new n and sig values in a download URL or JSON containing the download URLs, so the user can use an HTTP client directly, outside of Python. But the user still has to use Python to download each video page. Using an HTTP client directly would be faster.

Using yt-dl and yt-dlp just to output optimal download URLs feels like overkill.

If you find yt-dlp particularly slow, you can use aria2c to perform the download itself. It doesn't solve any startup latency problems, but it generally can do the HTTP requests faster.

This is complete nonsense. The overwhelming majority of video download time is spent transferring the actual video data, limited by throttling or network speed. The time it takes for these tools to start up and deal with the API/JS stuff is inconsequential for any video longer than a minute or so. And even then most of the time is network latency for the API stuff, not local processing. Netcat isn't going to go any faster.

Absolutely nobody thinks optimizing the meta/API processing of yt-dlp & co is worth it. This is exactly why we have high level programming languages that make all of this much easier, instead of trying to write HTML and JS parsing in plain C. Keep in mind these tools support dozens or hundreds or websites, not just YouTube.

If you think rewriting yt-dlp in C is worth it, go right ahead, but you're not going to make it significantly faster; you're just going to make maintenance a much bigger pain for yourself. Pick the right tool for the job. Python is absolutely (one of) the right tools for this job.

(For the record: I use C and Python on a daily basis, and will use whatever language is appropriate for each situation.)

"Python is absolutely (one of) the right tools for this job."

This opinion assumes it is a single job. I see multiple jobs. The number "options" provided by yt-dl(p) gives us a clue.

There is nothing wrong with preferring to use larger, more complicated, "multi-purpose" utilities. There will always be plenty to choose from.

However the idea of using smaller, less complicated, single purpose utilities is not "nonsense". It makes sense in many cases and some users may prefer it.

The statements I make about speed are from day-to-day experience not conjecture.

I look forward to seeing your implementation of this solution.

What is especially slow on some hardware like the original PinePhone is that youtube-dl and yt-dlp are actually executable that extract themselves at runtime. Python itself is quite fast to start.

The startup time for netcat is less than for yt-dl/yt-dlp.

Plus I can request multiple video pages over a single TCP connection with netcat.

For example, in a single TCP connection, with a list of 30 videoIds, I can get initial sig and n values for 413 videoIds. No wait time for netcat to decompress or startup. Using netcat is quite fast. Then I have utilties written in C to extract URLs from stdin. As such, all I need is a utility to update the sig and n values in the download URLs to make them fast ones instead of throttled.

How long would it take to get sig and n values for 30 videoIds let alone 413 with yt-dl or yt-dlp over a single TCP connection. The startup time for yt-dl/yt-dlp plus the time waiting for the downloading of each video page makes it much, much slower. These are "do-everything" scripts that are as a result quite inflexible.

Fixed typos/misspellings, edited for more clarity:

yt-dl and yt-dlp use Python to run the Javascript functions that modify the initial "n" and "sig" values to make non-throttled download URLs, but a faster language could be used to interpret and run the snippet of Javascript. For example, V8 is written in C++, not Python.

One of the reasons yt-dl and yt-dlp are so slow, IMO, is because they do (too) many other things besides running the Javascript, modifying n and sig, and spitting out a fast download URL. Before they can run the JS, the video page needs to be retrieved. I have self-created utilities for downloading webpages wth TCP clients that are sgnificantly faster and more flexible than yt-dl/yt-dlp. I use these small dedicated programs every day. I am used to the speed. Waiting for yt-dl/yt-dlp to start up in order to download web pages is annoying. The latency is unbearable.

Both yt-dl and yt-dlp have an option to output the new n and sig values in a download URL or as JSON containing the download URLs, so the user can use an HTTP client directly to download video, wthout need to launch yt-dl/yt-dlp. But the user still has to use yt-dl/yt-dlp to download each video page. Using a TCP client directly would be faster.

Using yt-dl and yt-dlp just to output optimal download URLs feels like overkill. It would be nice to have a program that just focuses on running the inecessary JS in base.js in order to output optimal download URLs. Then the user can use whatever programs she wants, directly, for downloading HTML/JSON, extracting URLs, downloading video files, converting video files, etc. Downloading video from most websites is generally easy. I never need a program like yt-dl/yt-dlp. It is only YouTube that plays games with users trying to get them to enable Javascript and be tracked. One need only look at the size of the extractors in yt-dl/yt-dlp as evidence. The extractor for YouTube is 3x the size of the next largest, and over 10-20x the size of most of the others. I want a small utility that just focuses on YouTube. A simpler solution instead of a massive project.

There is a pure js version here: https://www.npmjs.com/package/ytdl-core

Reply on news.ycombinator.com