Hacker News Reader

Comment by mpfundstein | Hacker News Reader

i like the function. saved me a lot of manual work already ;-)

just use it when you need it and be aware of the downsides...

And what if I'm searching through a large directory? Your response doesn't address the argument being made.

Creating an API where the entire set of data is returned from a data source, without an option to limit the result, is bad. It means that, if I have to read a directory with 2000 files (and this is quite possible! - think /bin, etc.) the function will have to allocate a slice with 2000 entries, and then call stat on all of them.

Stat is not free, especially on certain filesystems (NFS!), nor is reading directory entries.

that's why I wrote: be aware of the downside...

It's a mega convenient function if you want to slurp in a few files.

"ORM is great because it means I have all the data avaliable!"

try millions of files :). You can run out of inodes too :)

In what universe calling stat on every element of an array is “saving a lot of manual work”?

Yeah, it seems similar to `ioutil` functions which aren't necessarily the best approach in all cases, but are intended to make certain common tasks trivial for many cases.

Uh, k?

What does being aware of the downsides get you?

Is there some alternate interface that avoids the downsides?

I am just saying that the function has merit. Even if it could have been written much better. Thats all :-) It served me well.

Reg downsides: Maybe don't use it on a directory with 1000 entries...? Its good to be aware of that or not?

So just what the fuck are you supposed to use on a directory with 1000 entries?

You can drop down into the lower level apis https://godoc.org/golang.org/x/sys/unix

Yeah, nope. There are many appropriate levels of abstraction in between “do this the dumbest way possible” and “write raw unix syscalls using a library outside of the standard distribution”.

Just because something doesn't work for your use-case doesn't make it the "dumbest way possible". It just makes you sound dumb.

No, this is literally the dumbest way possible to accomplish the task. It is as close to a brute force approach as you can get, I can see no way in which it could be less efficient without being malicious. And it is the only way of accomplishing this task in the standard library of a language which prides itself on performance and maintainability.

Also, thanks for the ad hominem, perhaps you shouldn’t attach your self worth to a programming language and see how that works out for you.

There's https://golang.org/pkg/os/#File.Readdir

Which, as far as I can tell (couldn’t find opendir/closedir), has the same problem.

I think they named the call after the posix one. If so, they must have known about opendir/closedir.

It uses https://golang.org/pkg/syscall/#ReadDirent and https://golang.org/pkg/syscall/#ParseDirent to incrementally read them.

That interface is just awful, here's a byte buffer and some offsets, good luck.

https://github.com/karrick/godirwalk

Let’s not do participation trophies though, Pike and Thompson were the original authors of Go. Both of them should know better.

It’s fine to include an inefficient version if you also have a ‘proper’ way to fall back to when performance does matter. And if performance does not matter, why are you even using Go? Funnily enough Python Ruby and Node do expose performant ways to do this in the stdlib, so...

This is the syscall: https://www.man7.org/linux/man-pages/man2/getdents.2.html

You opendir, read 100 entries, do whatever with them, read 100 more, repeat, then closedir. So it's a buffered interface similar to normal I/O.

There's been proposals for doing readdir+stat in one syscall, but nothing merged yet. https://lwn.net/Articles/606995/

You can also use io_uring and at least do all the stats in a single syscall (or none, depending on what flags you create the ring with, but for normal use it would be one call).

Ah yes, forgot about uring. Apparently readdir was "next on the list" at the end of last year: https://twitter.com/mathjock/status/1205996402930855937 But I don't see it in the tree. I'm guessing sockets got more priority.

Reply on news.ycombinator.com