Hacker News Reader

Comment by enriquto | Hacker News Reader

For me the killer feature of wget is that by default it downloads a file with a name derived from the url.

You do:

    wget url://to/file.htm

and a file named "file.htm" appears in your cwd.

Using curl, you would have to do

    curl url://to/file.htm > file.htm

or some other, less ergonomical, incantation.

View on HN

    curl -O

https://curl.se/docs/manpage.html#-O

Also -OJ : with -O you get the name derived from the URL (the initial one, I think, even if redirects are being followed), with -OJ you get the one from the Content-Disposition header or the final URL, the way browsers do it. Of course, plain -O is safer. (For parity with Wget, you might also want to add -R to set the downloaded file’s mtime according to the Last-Modified header.)

Wget will give you an equivalent with the --content-dispisition flag. I would like for it to be the default, but it would likely break backwards compatibility with some scripts that except a different output filename.

-OJ is nice but doesn't work with -C- for some reason

True. AFAIU the reason is that Curl wants to make a single request (modulo redirects), whereas making -OJC- work would require two: issue a HEAD to receive the Content-Disposition header and learn the file name, then look at that file and see how long it is, then issue a GET with a Range header to request the suffix you still need to download. With other methods I don’t think you could make this work at all. I don’t know if Stenberg is opposed to a GET-specific solution, perhaps that could be a fun project. (Although I’ve encountered noncompilant servers that couldn’t handle HEADs.)

> curl -O

Yes. But the GP said by default.

No, OP said…

> you would have to do `curl url://to/file.htm > file.htm` or some other, less ergonomical, incantation

… which begs the conclusion that OP is unaware of `-O`.

I'm well aware of curl's -O (long form: --remote-name), but it has an unfortunate clash with an option of the same name from wget (long form: --output-document). These options have closely related yet very different meanings. Using these options without looking at the manpage fills me with a sense of dread that I prefer to avoid.

man curl | grep -C 2 " -O"

--remote-name-all This option changes the default action for all given URLs to be dealt with as if -O, were used for each one. So if you want to disable that for a specific URL after --remote-name-all has been used, you must use "-o -" or --no-remote-name.

alias curl='curl --remote-name-all'

I was obviously responding to the curl command they posted. I didn't say anything about default behaviour.

I much prefer the fact that curl doesn't do this by default (but has the option), it much closer matches the behavior of most unix-y tools. Makes it so much easier to put it into pipelines.

What's "better" depends on whether you 'just' want to download a file, or if you want to build an larger workflow.

For me, I usually want to download files and I'm usually not doing any more processing, so I tend to type in *wget" first.

I usually use `curl -OL <url>` or `curl -L -o <local-filename> <url> instead of wget.

wget is better for download lists, but I'd accept the argument that a simple shell script is similarly easy.

Yeah, but:

    wget "url://to/file.htm?uid=foo&q=bar&rnd=4"

`&unused=.htm`. It usually works.

At this point you might as well use the -o option (-o file.htm). It's easier and easier to understand.

I'd prefer wget to be a bit more clever when handling URLs query strings though, but I guess changing this behavior now might break some scripts.

The -O option, not the -o option. The capital O sets the output file, while the small o in your comment sets the log filename.

Yep, thanks for the correction. I meant big -O, I don't know how I ended up writing small -o.

well, depends on the usecase. sometimes you want the whole url, like when i want to mirror a site and it has stuff like foo.html?page=1 foo.html?page=2 ...

wget does have options to use the name proposed by the server, and so another option to remove the query arguments would be useful, and in line with those.

A new option to strip query parameters from the output filename would be interesting. But its not so simple. When combined with recursion, one will often see a lot of pages with the same name but different query parameters. How should they be stored on disk? There's a couple of different issues I can think of.

However, if the potential issues can be resolved with sane defaults, I think this would be a great new switch to add.

yes, exactly. i think that the option would have to be ignored when doing recursion. or alternatively use the .1 .2 ... method like with all cases where a file of that name already exists.

So... you're adding more noise to the filename?

What?

It's a simple solution to give the file the right extension, and preserving query parameters can be the right thing to do if you hit the same path repeatedly e.g. for pagination.

> It's a simple solution to give the file the right extension,

Oh, I see now.

Do you work with many tools that can't work with files if they don't have the "right" extension? I thought that was mostly a Windows problem.

I’ve always seen this as a misfeature of wget, on the general principle that command-line utilities should write their principal result to stdout unless otherwise instructed.

or curl -O which is more ergonomic

that "killer feature" for cat would be turn `cat file.html` into `cat file.html > file.html` which means if you actually wanted to cat instead of cp you'd also need `cat file.html -o -` kinda glad curl doesn't have that killer feature.

Reply on news.ycombinator.com