Xdm – Powerfull download accelerator and video downloader | Hacker News Reader

Xdm – Powerfull download accelerator and video downloader

github.com

41 pointsmaydemir45 comments

I used a lot of download managers and accelerators back in the dial-up times, they seemed like magic to me, plus being able to resume a download after somebody picked up the phone and dropped my internet connection was a plus!

How is it that they work, that is the accelerator part? My guess is an algo to find the optimal amount of parallel downloads?

Edit: also with parallel downloads, is it possible to do a HTTP GET with an bytes offset? With video I guess it’s possible to select what part to stream, but with regular http download I’m not sure

> an algo to find the optimal amount of parallel downloads

An algo...? Most download managers I tried simply had a set amount of threads they tried to open; the smartest ones will close some of those connections if they think they are being throttled, but that's rare.

That's just all there is to it: single-threaded downloads are severely inefficient in the age of multithreaded CPUs (and of operating systems without a finely-tuned network stack), so by leveraging parallelization anywhere you get a chance, you multiply the speed.

> single-threaded downloads are severely inefficient in the age of multithreaded CPUs

Is the CPU typically the bottleneck when downloading things from the Internet?

Nope. Starting at the NIC, it's not "single-threaded." Intel NIC drivers on most platforms service the multiple buffers using several available physical cores. Even if it were single-threaded, memory bandwidth over a decade ago was around 3GB/sec.

Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends.

Download "managers" that can queue up a list of downloads, schedule downloads (say for when you're asleep), automatically retry/partial-resume a big download, throttle a download to a low rate, and so on? That's useful for some folks.

Edit: it's written in java. I'm laughing my ass off.

> Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends.

If there are any network bottlenecks, including at the host, that are enough to stop you saturating your own last-mile link with a single stream, that is simply not true. Or if the sending end sets per-stream limits. Downloading multiple streams from the same source can significantly speed up the process. Maybe it often doesn't, but that doesn't make it snake-oil.

> memory bandwidth over a decade ago was around 3GB/sec

Memory bandwidth is practically irrelevant if the main bottleneck is the remote network or something between you and it.

If the app uses a single execution thread, the OS will likely be unable to distribute work across different CPU threads. Whereas if you create multiple threads, the OS can max out CPU and I/O resources efficiently.

but with today CPUs this becomes an issue with speeds at least 25gbps

Yeah.

Twenty year old CPUs had 1600MB/sec memory. 4-5 years later memory speed doubled. In 2010 it doubled again.

The Ryzen in my current desktop has memory that tops out at 25GB/sec.

> the smartest ones will close some of those connections if they think they are being throttled, but that's rare.

And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).

You want to use the Range header.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ra...

Point in this project's code where you can see this happen [0].

Also interesting to note that it's not just HTTP it supports but other protocols I haven't heard of like DASH, HDP, HLS [1].

[0] https://github.com/subhra74/xdm/blob/master/app/src/main/jav...

[1] https://github.com/subhra74/xdm/tree/master/app/src/main/jav...

Those protocols are on top of HTTP so it makes sense they support it too.

so that's how `wget -c` works! the more you know :)

Since flags are hard to google, what does `wget -c` do?

My favorite way to find out what flags do is explainshell:

e.g. https://explainshell.com/explain?cmd=wget+-c

Wow that's a cool idea! I'll have to try it out next time.

Resume partial download.

Gotcha, thanks

Last code commit in December 2020: https://github.com/subhra74/xdm/commit/953765940ae63107e8cfb...

Apparently it is being rewritten from Java to .Net: https://github.com/subhra74/xdm/discussions/768

One tool I remember using a lot of was "DownThemAll". It was great to download quickly using a regex for files.

I recommend aria2 for a similar purpose. Just `apt install aria2`. I'm just a happy user.

+1 for aria2! It does one job and does it very well.

Want to add a UI? Take a look at WebUI-Aria2: https://github.com/ziahamza/webui-aria2

It is also very easy to integrate with othersoftware via the RPC interface https://aria2.github.io/manual/en/html/aria2c.html?highlight.... You can find wrappers in some languages e.g. the JS bindings: https://github.com/sonnyp/aria2.js

Together with the browser extension, I've successfully used it to download hundreds of videos. There are many other popular video downloader browser extensions, but often they contain spyware. Being open source gives me slightly more confidence in XDM.

I uses Video DownloadHelper (Firefox addon) with JDownloader2 (Java standalone app), I only use Video DownloadHelper to grab the video header and send it off to JD2 to do the download and conversion (via FFmpeg).

The video platform that I couldn't find a way to download is JW Player. I tried various extensions (Firefox/Chrome) and standalone apps, they couldn't download the video stream from JW Player. The only thing they could grab is m3u/m2u8 file and that it.

Nice to see there are another Java downloader.

How high are the chances it uses a yt-dlp backend?

It written in Java, so while not impossible, I'd say unlikely.

The UI has always seemed "off." I'll stick to Free Download Manager

How does this compare to youtube-dl or yt-dlp?

So not X Display Manager?

* https://en.wikipedia.org/wiki/XDM_(display_manager)

* https://en.wikipedia.org/wiki/X_display_manager

Not sure the Latin alphabet is wide enough for all the acronyms we need in recent years. :)

Edit: reminds me of a story. Supposedly (!) the reason "Xkcd" is named so is because at the time of creation there were no Google results for that letter combination.

I snagged a 4 letter domain name about 10 years ago and will never abandon it. Even if it's a .org

hmm, a.org is incubating the future, good snag.

It's a shame that b.org has been domain squatted since 1993. Not even a placeholder website.

I wonder how much it would cost to assimilate that site.

"Unix is running out of two letter acronyms."

Software can't give you faster download speeds. If I get 10 Mbps down from my ISP I'm not going to somehow download at 50 Mbps. yt-dlp, steam, and firefox already max out my connection when they download something.

To nitpick: Software can give you faster effective download speeds. Compression is software. Parallelization across multiple hosts is achieved in software.

It really depends on the cause of the bottleneck. Sometimes you need capabilities on both ends though.

Also, while I'm nitpicking: "xdm" is and will always be the X Display Manager. :(

The biggest problem facing technology in the early 22nd century is that there are only 17,576 TLAs, and they're all overloaded already.

That’s why we also use ETLAs

I've ahead of the game, I jumped directly to FETLAs.

What they _might_ mean to say is that it gets around throttling at the source, probably by attempting to download multiple pieces in parallel.

The bottleneck is typically at the server level, not the pipe.

Steam uses BitTorrent, which is built to parallelize downloads - effectively implementing what download managers do. I think Firefox these days also does clever stuff behind the scenes. The result is downloads that complete faster than before, even though the theoretical max speed of any given connection is the same.

Download managers are somewhat old-fashioned simply because modern software learnt from them and now tries their tricks out of the box.

>Steam uses BitTorrent,

No, it downloads encrypted chunks from Steam's CDN.

yt-dlp supports this sort of thing too..

> Multi-threaded fragment downloads: Download multiple fragments of m3u8/mpd videos in parallel. Use --concurrent-fragments (-N) option to set the number of threads used

Not all sites are willing to max out your pipe with a single connection. Making multiple connections to those sites often gets around that.

It can't get you faster than your ISP's throttling, but it can get you faster than the server you download from's throttling.

I've been stuck with a throttled 10mbit fiber connection before, and still had times where sites would throttle me more than that. Now that I have a gigabit connection, it's a much more common problem.

Sometimes the bottleneck is on the server end, and often they throttle throughput per connection. Some download mangers allow you to download the same file through multiple connections and range request. Having said that I think it's a little bit rude to work around the intended throughput limitation.

Reply on news.ycombinator.com