Hacker News Reader

Comment by _benj | Hacker News Reader

I used a lot of download managers and accelerators back in the dial-up times, they seemed like magic to me, plus being able to resume a download after somebody picked up the phone and dropped my internet connection was a plus!

How is it that they work, that is the accelerator part? My guess is an algo to find the optimal amount of parallel downloads?

Edit: also with parallel downloads, is it possible to do a HTTP GET with an bytes offset? With video I guess it’s possible to select what part to stream, but with regular http download I’m not sure

View on HN

> an algo to find the optimal amount of parallel downloads

An algo...? Most download managers I tried simply had a set amount of threads they tried to open; the smartest ones will close some of those connections if they think they are being throttled, but that's rare.

That's just all there is to it: single-threaded downloads are severely inefficient in the age of multithreaded CPUs (and of operating systems without a finely-tuned network stack), so by leveraging parallelization anywhere you get a chance, you multiply the speed.

> the smartest ones will close some of those connections if they think they are being throttled, but that's rare.

And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).

> single-threaded downloads are severely inefficient in the age of multithreaded CPUs

Is the CPU typically the bottleneck when downloading things from the Internet?

If the app uses a single execution thread, the OS will likely be unable to distribute work across different CPU threads. Whereas if you create multiple threads, the OS can max out CPU and I/O resources efficiently.

but with today CPUs this becomes an issue with speeds at least 25gbps

Yeah.

Twenty year old CPUs had 1600MB/sec memory. 4-5 years later memory speed doubled. In 2010 it doubled again.

The Ryzen in my current desktop has memory that tops out at 25GB/sec.

Nope. Starting at the NIC, it's not "single-threaded." Intel NIC drivers on most platforms service the multiple buffers using several available physical cores. Even if it were single-threaded, memory bandwidth over a decade ago was around 3GB/sec.

Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends.

Download "managers" that can queue up a list of downloads, schedule downloads (say for when you're asleep), automatically retry/partial-resume a big download, throttle a download to a low rate, and so on? That's useful for some folks.

Edit: it's written in java. I'm laughing my ass off.

> Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends.

If there are any network bottlenecks, including at the host, that are enough to stop you saturating your own last-mile link with a single stream, that is simply not true. Or if the sending end sets per-stream limits. Downloading multiple streams from the same source can significantly speed up the process. Maybe it often doesn't, but that doesn't make it snake-oil.

> memory bandwidth over a decade ago was around 3GB/sec

Memory bandwidth is practically irrelevant if the main bottleneck is the remote network or something between you and it.

You want to use the Range header.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ra...

so that's how `wget -c` works! the more you know :)

Since flags are hard to google, what does `wget -c` do?

Resume partial download.

Gotcha, thanks

My favorite way to find out what flags do is explainshell:

e.g. https://explainshell.com/explain?cmd=wget+-c

Wow that's a cool idea! I'll have to try it out next time.

Point in this project's code where you can see this happen [0].

Also interesting to note that it's not just HTTP it supports but other protocols I haven't heard of like DASH, HDP, HLS [1].

[0] https://github.com/subhra74/xdm/blob/master/app/src/main/jav...

[1] https://github.com/subhra74/xdm/tree/master/app/src/main/jav...

Those protocols are on top of HTTP so it makes sense they support it too.

Reply on news.ycombinator.com