os.Readdir has to be one of the stupidest, most broken interfaces I've ever seen.
Reading the whole dir into an array is bad enough as it is, but they even call stat() on every file too.
Reading the whole dir into an array is bad enough as it is, but they even call stat() on every file too.
While I can't speak for Go's authors' mindset, here's my view:
One array of records is about as simple interface as possible, and it's good enough for many use cases. Contrasting with that, reading the directory with repeated readdir()'s or equivalent is fraught with hidden pitfalls. You end up being responsible for sorting, for resuming syscall in case of a signal or other interruption, for handling changes to the list of files during read... and probably other concerns above and beyond that.
If an API is going to arbitrarily sort filenames, at least let us pass in a comparator function so we have some control. That would actually be a decent improvement to its API.
os.Readdir() fetched 10.000 file names into a slice in 0.2s.
That's more than fine for most use cases and certainly doesn't look like "one of the stupidest, most broken interfaces I've ever seen".
Specially given that it's not the only way to iterate over files of a directory in Go. There's filepath.Walk() for one.
> The files are walked in lexical order, which makes the output deterministic but means that for very large directories Walk can be inefficient.
If it's fine for "most use cases," that's all well and good. Just implement the correct interface, which is fine for ALL use cases, and then implement this sugar on top of that.
Seems like they postponed incorporating fastwalk into the standard library because it would break API: https://godoc.org/golang.org/x/tools/internal/fastwalk
Perhaps until then one could use libraries that expose fastwalk:
I now built the exe and ran:
real 0m0.199s
user 0m0.000s
sys 0m0.014s
It is still a ton of time, about an order of magnitude more than optimal if the real/sys time split is to be believed.
It's not a "stupid" interface if it does exactly what you need. And I think this is a pretty common use case.
Otherwise, you'd better be 110% sure of this:
> Even when you know that you're dealing with a small number of files, and performance isn't critical?
Because if you turn out to be wrong down the line, your only option is now to rewrite the whole thing in a less-stupid language. Might as well just save yourself the trouble and do that from the beginning.
just use it when you need it and be aware of the downsides...
What does being aware of the downsides get you?
Is there some alternate interface that avoids the downsides?
Reg downsides: Maybe don't use it on a directory with 1000 entries...? Its good to be aware of that or not?
Also, thanks for the ad hominem, perhaps you shouldn’t attach your self worth to a programming language and see how that works out for you.
I think they named the call after the posix one. If so, they must have known about opendir/closedir.
It’s fine to include an inefficient version if you also have a ‘proper’ way to fall back to when performance does matter. And if performance does not matter, why are you even using Go? Funnily enough Python Ruby and Node do expose performant ways to do this in the stdlib, so...
You opendir, read 100 entries, do whatever with them, read 100 more, repeat, then closedir. So it's a buffered interface similar to normal I/O.
There's been proposals for doing readdir+stat in one syscall, but nothing merged yet. https://lwn.net/Articles/606995/
Creating an API where the entire set of data is returned from a data source, without an option to limit the result, is bad. It means that, if I have to read a directory with 2000 files (and this is quite possible! - think /bin, etc.) the function will have to allocate a slice with 2000 entries, and then call stat on all of them.
Stat is not free, especially on certain filesystems (NFS!), nor is reading directory entries.
It's a mega convenient function if you want to slurp in a few files.