Hacker News Reader

Comment by throwaway76543 | Hacker News Reader

> There's nothing really like it anymore - you can't have anonymous sftp

Nonsense. http is exactly like anonymous ftp and it does a much better job of it. Pretty much every anonymous ftp site started also serving their files via http decades ago -- which is why ftp is no longer needed.

Case in point: Debian makes all these files available over http. This isn't going away.

ftp://ftp.debian.org/debian/

http://ftp.debian.org/debian/

View on HN

It really isn't as convenient if you have to download lots of files at one time though. FTP has mget. That's probably why FTP lives on for scientific data (NCBI, ENSEMBL, etc). Yes, you could use some tool like wget or curl to spider through a bunch of http links, but that's more work.

I suppose you can sometimes do `mget x.csv` on FTP (if the client supports it?), but with wget you can do:

wget -r -A 'x.csv' https://example.org/

(where 'x' is an asterisk, but HN's formatting eats it)

More work, in the sense that it's more command line options to remember, I agree, but otherwise it's easier to integrate in scripts and much more flexible than mget.

(I don't miss FTP for the sysadmin side of maintaining those servers.)

> FTP has mget

Not quite, ftp CLIENTS have mget. The ftp protocol has absolutely no awareness of mget. In fact, ftp is terrible at downloading more than one file at a time because it has no concept of pipelining and keepalive, both things that http supports.

With a nice multi protocol client like lftp, http directory indexes work just like an ftp server:

  $ lftp http://http.debian.net/debian/
  cd: received redirection to `http://cdn-fastly.deb.debian.org/debian/'
  cd ok, cwd=/debian
  lftp cdn-fastly.deb.debian.org:/debian> ls
  drwxr-xr-x  --  /
  -rw-r--r--         1.0K  2017-01-14 10:44  README
  -rw-r--r--         1.3K  2010-06-26 09:52  README.CD-manufacture
  -rw-r--r--         2.5K  2017-01-14 10:44  README.html
  -rw-r--r--          291  2017-03-04 20:08  README.mirrors.html
  -rw-r--r--           86  2017-03-04 20:08  README.mirrors.txt
  ..[snip]..
  lftp cdn-fastly.deb.debian.org:/debian> mget README*
  5315 bytes transferred
  Total 5 files transferred
  lftp cdn-fastly.deb.debian.org:/debian>

Wow, I had no idea lftp had that feature. That's super cool.

lftp has a ton of features. background jobs, tab completion, caching of directory contents, multiple connections, parallel fetching of a SINGLE file using multiple connections.

Yes, it looks like '/usr/bin/ftp' from 1970, but it's far far far more advanced than that.

For anyone else wondering how to recursively download with this:

    lftp cdn-fastly.deb.debian.org:/debian> mirror doc
    Total: 2 directories, 43 files, 0 symlinks                             
    New: 43 files, 0 symlinks
    1031755 bytes transferred in 1 second (678.7 KiB/s)
    lftp cdn-fastly.deb.debian.org:/debian>

Warning, -R means reverse (upload!), not recursive. ;)

Download managers such as Down Them All! [1] for Firefox are more convenient (and useful) than mget in an ftp client.

[1]: http://www.downthemall.net/

It would make more sense to offer rsync (unauthenticated), to ensure integrity of what's transferred.

  rsync -r rsync://... ./

will retrieve everything in a directory.

Reply on news.ycombinator.com