Hacker News Reader

Comment by nicoburns | Hacker News Reader

The article mentions that Go, OCaml, or a compiled lisp were considered for this project. I wonder why Rust wasn't on that list. It seems to cover every concern that is raised here except for keyword arguments (which are high on my list of desired Rust features too).

I guess maybe they were worried about the lack of GC, but my experience has been that's it's generally quite easy to port code from dynamic languages like Python/JS/PHP to Rust. So I suspect that may have gone into their "Expected problems that weren’t" section if they'd tried it.

View on HN

Async wasn’t in yet at the time. He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go.

Incidentally, I helped with the port to Go, and I spent some time polishing the code, and finding and fixing performance problems. When we were helping GCC convert their SVN repository to Git (a repository with 287k commits, btw), we reduced the memory usage by 50% (from over 250GB to under 128GB), and the run time by quite a lot as well (down to just around 2 hours to read in the SVN repository and convert it to a basic Git repository).

Now that we’ve done that work, Reposurgeon spends 50–60% of its cpu time scanning the heap for garbage. There is often garbage to find, but just as often there is not. GC is useful, but for Reposurgeon it has become a bottleneck.

My preliminary work on a Rust port shows that it is around 4× faster than the Go version. I personally think that Rust is the future, but I haven’t been able to put as much effort into the port as I would like.

Appreciate your thoughtful reply! One question. Regarding this bit:

> He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go.

Is this stating that he was writing the Rust version similarly to how he would have written it in C because that was the most natural way to do it in Rust or simply because it was the first path he went down? In other words, was your point that Rust is a fundamentally poor fit for this problem, or was it that his lack of experience with the language took him down the wrong path?

Thanks!

He went down the wrong path, but at the same time Rust didn’t do anything to make async code easier to write. You just had to call libc::select and std::thread::spawn and whatnot, with all of the unsafe blocks that implies. These days there are about 47 different crates that can help you do it, all of them a lot nicer than that.

That's what I thought, but I just wanted to confirm. Thanks!

You’re welcome.

Here is what esr wrote about Rust in early 2017:

http://esr.ibiblio.org/?p=7294

http://esr.ibiblio.org/?p=7303

> A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying.

This is exactly the impression I have gotten from the crates system every time I've looked. It happens for things as foundational as mmap. Also terrifying: The plethora of highly-recommended crates that have not yet committed to a stable API (ie, are still semantically on version 0.X).

I’m not sure that’s a great example. mmap(2) is a swiss army knife of a function. There are many crates that build all kinds of things on top of mmap, and they’re not all interchangeable. Allocators, file io, gpio, actual virtual memory mappings.

My view is that when you develop a product, you have to own everything. The users don’t care if the bug comes from code you wrote, or code in a library, or the language’s standard library, or the OS. You have to fix the bug no matter what caused it. It doesn’t matter if the code came from a vendor or a language designer or stack overflow; you are the one responsible for fixing it if something goes wrong. Non est salvatori salvator, etc.

With that perspective, I don’t think that 23 crates is terrifying. I’m going to look them all over and either pick one, or write the 24th crate myself. The result is the same either way.

I do agree with you about APIs though. It is nice to find a crate where the author has had the confidence to stabilize their API and declare the version to be 1.x instead of 0.x. But at the same time I recognize that getting to that point requires some real software to use the crate, to create the feedback loop. If nobody ever used a 0.x crate, no crate would ever get that feedback.

>My view is that when you develop a product, you have to own everything. The users don’t care if the bug comes from code you wrote, or code in a library, or the language’s standard library, or the OS. You have to fix the bug no matter what caused it. It doesn’t matter if the code came from a vendor or a language designer or stack overflow; you are the one responsible for fixing it if something goes wrong.

So then using a programing language that has a big standard library and a more rich and stable ecosystem is a big advantage. If I want to do a network request, parse a json file, parse or format a date or some other trivial thing I prefer to have a good enough built in way to do it then evaluate 15 packages or write it myself. Otherwise you get in a shitty situation where you inherit some projects and it has 20 dependencies with vulnerabilities, 20 dependencies abandoned, some dependencies that are incompatible with some new version of the language/ecosystem causing issue if you would like to upgrade.

Sometimes. Using someone else’s code is only a benefit if it saves time overall, compared to writing it yourself. You have to count both the time to write it and the time to maintain it over however long your product is active. You have to assume that you’ll be maintaining it yourself either way.

Also, don’t forget that languages with big standard libraries, like Python or C# or Java, always end up with oodles of stuff in there that is deprecated or incompatible. That’s as bad a sign as anything.

But I do agree with you, a lot of the time you have inherited the mess rather than writing it yourself. Not much you can do about it, except to clean it up and then dodge better next time. Going back to Rust specifically, Cargo gives you some tools that make the mess easier to clean. The Rust language also helps, because you can safely use multiple versions of your dependencies, when it turns out to be necessary.

>Also, don’t forget that languages with big standard libraries, like Python or C# or Java, always end up with oodles of stuff in there that is deprecated or incompatible. That’s as bad a sign as anything.

This does not happen that much, and in Java and .Net world if some functionality is deprecated there is a replacement, I think I only remember some possible unsafe or ineeficient functions were deprecated so you use the better ones. I don't have experience with Rust , only with node/npm and is a hell , for some reason is decided to split stuff in super small pakcagtes of various quality, today I sepnd hours debugging a npm freeze caused by some package , in the end the cause is probably a shit npm/node implementation that crashes when some specific git version is installed , but debugging this I discovered that the unit testing packages the project uses(I inherited them) instead of beeing one or few packages are a few, and for some reason one of them had a weird dependency that it should not have(a dev only despondency ) and this dependency was also just pulling directly from GitHub ...shit in a few years when GitHub is gone lots of things will stop working (I also had issues with scripts because someone changed master into main because American politics...).

I personally prefer the Java or .Net model, a big standard lbirary and then for most important things you have a small number of options since the community did not wanted to do CV driven development, and most of the time you find all you need in a single library with no or few dependencies. But Rust does not have the Java or .Net money to hire devs to work on the boring stuff of correctly implementing standards like json,XML, date&time and keep maintaining it - (not sure how Python did it) and Rust community seems to be inspired byt node community a lot and this is bad.

> With that perspective, I don’t think that 23 crates is terrifying. I’m going to look them all over and either pick one, or write the 24th crate myself. The result is the same either way.

In a fine-grained ecosystem, the problem is transitive. You aren't just picking one from 20+ for each of your direct dependencies, you are also trusting that they in turn did just as much diligence as you did in picking their direct dependencies and so on. Curation and pruning would at least help mitigate the scope of the problem.

> ... or write the 24th crate myself.

This is a good example of the xkcd joke about standards proliferation. If I found myself in this situation, I'd prefer to keep the 24th private to my own package. DRY has its limits. Sometimes you've just got to specialize.

I'm aware of the irony in saying this in the context of mmap. But in this case, a thin wrapper that reflects the system call's semantics ought to be available in a common `posix` crate that others may freely rely upon to build their higher-level services. There's no good reason for e.g. ripgrep to have to pick and choose among a dozen flowers for it.

> In a fine-grained ecosystem, the problem is transitive.

Absolutely. You definitely have to own the whole stack. Ultimately, you may have to fix bugs at every level, from your own code all the way down to the OS. Big tech companies do this explicitly, with kernel development teams. Small companies do this by occasionally upgrading to the latest version of Ubuntu and hoping for the best.

> But in this case, a thin wrapper that reflects the system call's semantics ought to be available in a common `posix` crate that others may freely rely upon to build their higher-level services.

It is available. If you want to directly call any function in the C standard library, use the libc crate. Here’s the documentation: https://docs.rs/libc/0.2.109/libc/fn.mmap.html

All the other crates on crates.io that you find when you search for “mmap” are higher level abstractions over the raw syscall. They have specific purposes, like file io, or creating circular buffers.

I agree, but at the same time things tend to stabilize over time as the community settles around the best packages. The real thing to fear is the relative instability of Rust at the present moment (for certain things, like async), but the overall trajectory seems promising.

It sounds that it was hard to learn and use for him, that's why he preferred Go and Python. It's a frequent complaint and the step from Python to Go is indeed easier than from Python to Rust. He also had some philosophical concerns about the crate system.

Two threads from back in the day about this:

https://news.ycombinator.com/item?id=13385530

https://news.ycombinator.com/item?id=13398068

Go, OCaml and Lisp (depending on the Lisp) all have one other thing in common.. a fast compile/feedback loop. Rust doesn't have a good story here. This would be my guess as to why (at least in part). Particularly coming from a dynamic languages which have super fast compile times if they compile at all.

Reply on news.ycombinator.com