Hacker News Reader

Comment by steveklabnik | Hacker News Reader

I'm not sure what you mean by not understanding it. It is true that if you don't have dependencies, "cargo build" will go get them, but if you don't, then it shouldn't. There have been some bugs that caused it to over-download, but those were bugs.

  > happily downloaded a random pre-built ELF binary to build itself.

Yes, Cargo builds itself with Cargo, so you need to get a previous Cargo. The binary is very much not 'random'.

View on HN

> It is true that if you don't have dependencies, "cargo build" will go get them, but if you don't, then it shouldn't.

The main problem is you can't download dependencies separately from compiling your code. It should never be the same step. Even pip had it mostly correct, by providing an option to disable network communication.

It is somewhat mitigated by documentation mentioning where the downloaded dependencies are put, so you can do all that manually. This sucks balls heavily.

> Yes, Cargo builds itself with Cargo, so you need to get a previous Cargo. The binary is very much not 'random'.

It's fetched outside of regular downloading you do on tarball. You have no control over where is it downloaded from. It means it's pretty much random for your purposes. Heck, it would be even unexpected, if I wasn't used to developers not understanding the difference between building and downloading software.

But this is not the main point. Cargo should not need already compiled Cargo to build itself. This is ridiculous. I understand that every good compiler is built with the very same compiler you're building, but look at how they do that: by bootstrapping. Not so much with Cargo. Who ever thought it as a valid idea?

  > It should never be the same step.

This is just going to be a preference then. I much prefer not having to type two things to do what's conceptually one: please build my code.

"cargo fetch" means you don't have to do the download step manually. IE, you _can_ do it in two commands if you'd like.

  > It means it's pretty much random for your purposes.

We are just going to have to agree to disagree on this one. First of all, you _already_ need an equally 'random' binary: rustc itself. Which is also bootstrapped. Second, it's not clear to me why bootstrapping is valid for a compiler, but not for any other kind of project. Cargo is a build system, written in Rust, so it uses Rust's build system.

> This is just going to be a preference then.

No, it is not.

* It's not always that you have internet connection when building your code.

* It's not always that you have access to the website code is downloaded from (websites break, and companies have different policies about internet), and even if you do, you don't always have it direct (e.g. proxy).

* And at the last, people will build code for various purposes, including building packages. Building a package should be a repeatable operation and should always use only clean source (no artifacts from previous builds), and downloading random things on every build from clean source is very easy way to break the process.

>> It means it's pretty much random for your purposes.

> We are just going to have to agree to disagree on this one. First of all, you _already_ need an equally 'random' binary: rustc itself.

No. I downloaded Rust compiler myself (or had it installed from a package). I controlled that, it didn't hit me in my face unexpectedly.

> Which is also bootstrapped.

s/also//. Cargo is not bootstrapped, because it needs pre-built Cargo to build itself. Compare this to Rust compiler: you start with nothing but C(++? I don't remember) compiler, and end with Rust compiler. No intermediate download involved.

> Second, it's not clear to me why bootstrapping is valid for a compiler, but not for any other kind of project. Cargo is a build system, written in Rust, so it uses Rust's build system.

It's not invalid, quite the contrary. It's just Cargo doesn't boostrap itself out of clean code. It's that simple. I wouldn't have as big problem with it if its build process produced an intermediate, crippled Cargo binary.

I would then complain about requiring external dependencies to be downloaded (in contrast to included), but that would be difference on strategy preferences.

Going to have to agree with dozzie here, he's right on almost all accounts.

There's a ton of things I like about Rust but the points he outlined above are things that I've seen very much impede the adoption of Rust into some environments that have similar restrictions(no download on build, build from source without binaries) for security and reproducibility reasons.

As I said to the parent, if offline builds do not work, please file bugs. Integrating Rust code into Firefox requires the same restrictions that you're talking about, and is a use case we explicitly want to support, and already do, to a certain extent.

> There's a ton of things I like about Rust but the points he outlined above are things that I've seen very much impede the adoption of Rust into some environments that have similar restrictions(no download on build, build from source without binaries) for security and reproducibility reasons.

They're all on the roadmap for the very near future, because Servo (and Rust-in-Gecko) needs them. There's nothing in the design of Cargo that prevents these from being solved.

Personally, I love Cargo. Autoconf is miserable and I would hate to go back to it.

Reproducibility is the thing that bothers me most about auto-downloading build systems. On the other hand, Cargo (IIRC) doesn't do what Maven does: if you say you want version X, you get version X or anything newer (unless you jump through hoops that I've never seen anyone jump through).

[Security just scares the piss out of me in general; downloading binaries is probably the smallest problem.]

Cargo follows semver by default. Specifying version "1" means "1.x.x, <2".

Pinning versions is simple. e.g., "=1.2.3".

You can read more about it here: http://doc.crates.io/crates-io.html

And even when you have a range, Cargo.lock means that you get reproducable (in this sense) automatically.

A few things:

1. If an offline build does not work, that's a bug. This use case is one that we explicitly want and need, and do already support.

2. You are incorrect about Rust. Rust is written in Rust, not C++, and so building Rust involves downloading a binary of a previous Rust. That's what 'boostrapping' means.

I thought that rustc was boostrapped from the old c based compiler(allowing you to build clean from source) multiple times(C -> Rust 0.x? -> Rust 1.0)?

From what I last saw of cargo there's not a clean path that lets you build from source that doesn't involve pulling binaries(and is really painful if you want to use cargo on a non-binary platform like RPi.

There's never been a C-based compiler. Rust started out as OCaml. A true bootstrap would take weeks of non-stop compiling. You'd end up compiling rustc itself about 900 times, ignoring all the different LLVM builds, I think. Rust bootstraps from a binary snapshot of itself, and so you would have to work through the list of every snapshot ever (about 300, but you need to build rustc 3 times for each snapshot to properly bootstrap that copy). The snapshots are listed here: https://github.com/rust-lang/rust/blob/master/src/snapshots....

In the past the snapshots were taken almost weekly as massive language churn dictated. Now they're quite rare.

The situation with Cargo would be exactly the same: before Cargo existed, it couldn't use Cargo, because Cargo didn't exist. You can boostrap your own Cargo from that time period if you'd like, but that doesn't mean it's easy. But it's no different than rustc; where you can bootstrap your own snapshots from the OCaml compiler.

> I thought that rustc was boostrapped from the old c based compiler(allowing you to build clean from source)

What's with all the baseless assumptions of the initial Rust compiler being written in either C or C++ in this thread?

Contrary to apparently popular belief, C and C++ aren't some primordial lifeform compiler language which all language implementations have to be initially written in. Believe it or not, you can write a compiler in any (general purpose) programming language! Amazing, I know.

1. Funny. Last time I built Cargo it insisted on downloading Cargo, and I've never seen even a hint that it can build itself without that. I needed to download dependencies manually, compile them manually, and compile Cargo's code manually as well.

> 2. You are incorrect about Rust. Rust is written in Rust, not C++,

And how that renders me incorrect? GHC is written in Haskell, but you can build it with C compiler (and then recompile again, now with Haskell compiler).

But indeed I was wrong. I remembered incorrectly that I built Rust cleanly and without network activity. I actually downloaded sources and issued compilation command, without any regard for packaging, so it could do any stupid stuff like downloading things when compiling.

> and so building Rust involves downloading a binary of a previous Rust. That's what 'boostrapping' means.

No. "Bootstrapping" means building a compiler using a compiler when you don't have that compiler. As I said elsewhere, compare that to OCaml and GHC, which can be built without downloading OCaml or GHC, respectively.

I don't think we're going to come to any sort of consensus here, so I'm going to drop off. Have a good day!

> * It's not always that you have internet connection when building your code.

Cargo does not require this.

> * It's not always that you have access to the website code is downloaded from (websites break, and companies have different policies about internet), and even if you do, you don't always have it direct (e.g. proxy).

Supporting local proxies is on the Cargo roadmap.

> Compare this to Rust compiler: you start with nothing but C(++? I don't remember) compiler, and end with Rust compiler. No intermediate download involved.

This is incorrect. Rust has been self-hosting for years. Before it was bootstrapping, it was written in OCaml.

> It's not invalid, quite the contrary. It's just Cargo doesn't boostrap itself out of clean code. It's that simple. I wouldn't have as big problem with it if its build process produced an intermediate, crippled Cargo binary.

Cargo is basically just part of the Rust compiler. Rust needs Rust to bootstrap itself, like tons of other languages. So Cargo needs Cargo to bootstrap itself. This really isn't a problem.

The fact that you didn't even know that Rust is self-hosted is proof that it really doesn't matter to the user—it was so invisible you didn't notice it!

>> Compare this to Rust compiler: you start with nothing but C(++? I don't remember) compiler, and end with Rust compiler. No intermediate download involved.

> This is incorrect. Rust has been self-hosting for years.

Incorrect in that Rust actually downloads something, as steveklabnik pointed. I admit that I expected more from Rust, and I feel disappointed the same with the compiler as with Cargo.

But your premise that self-hosted language automatically means that bootstrapping can't be done using C or C++ is false. Compare that with OCaml and GHC.

> The fact that you didn't even know that Rust is self-hosted is proof that it really doesn't matter to the user—it was so invisible you didn't notice it!

It could be tried to read as that, if it was a fact. I did expect Rust to be self-hosted, just it is a custom that bootstrapping compiler is written in something else.

Look at how OCaml or GHC (Haskell) are built. Both allow to use C compiler to compile them (even though they generally advise to use pre-compiled compilers).

> Both allow to use C compiler to compile them (even though they generally advise to use pre-compiled compilers).

How many people actually do this, though?

We had this conversation early on in Rust's life and the consensus was that we could make a non-bootstrapping compiler in theory, but that'd be asking us to do a huge amount of work (writing a separate compiler!) for something that very few people are going to use in practice. There are so many more important things that we could be (and are) working on than something that's basically just for purism, because most people just "apt-get install rust" or "brew install rust" and don't care how it's built.

> How many people actually do this, though?

Enough to warrant this being kept after eight years after I've seen it for the first time.

> We had this conversation early on in Rust's life and the consensus was that we could make a non-bootstrapping compiler in theory, but that'd be asking us to do a huge amount of work (writing a separate compiler!) for something that very few people are going to use in practice.

Note that it's not necessary to have fully blown compiler. Just a subset of language would be just enough, if it allowed for compiling the compiler. And how many of the Rust features are embedded in rustc, anyway?

> There are so many more important things that we could be (and are) working on than something that's basically just for purism, because most people just "apt-get install rust" or "brew install rust" and don't care how it's built.

So you basicaly say "fuck you" to distribution developers and all the sysadmins that care about their systems, am I right?

> So you basicaly say "fuck you" to distribution developers and all the sysadmins that care about their systems, am I right?

Your aggression, insulting tone and condescension (throughout your comments in this thread) is not appreciated. Please stop.

> And how many of the Rust features are embedded in rustc, anyway?

All of them. This is not an exaggeration.

Where are you getting your C compiler from?

"Before it was bootstrapping, it was written in OCaml."

That's been my exact recommendation for getting compilers started in robust way. You people keep surprising me in pleasant ways. :)

Curious, did Ocaml's clean syntax and ease of decomposing functions make the transition to Rust easier vs a language like C++ or Java? I predicted ML languages should have that benefit but I couldn't test it on small projects.

Back when I was in the university (90's), Lisp, Prolog and Caml Light (OCaml's percursor) were forbidden to be chosen in compiler design classes, although we used them a lot in other classes.

The reasoning for the rule, was that they made our compiler assignments too easy.

On my case, we ended up using the recently introduced Java with JavaCC and NASM macros for the generated bytecode.

Nowadays I would only use something like C++ for the runtime library, or reuse of existing tools, e.g. LLVM.

That's so funny. More support for my recommendation. Since you mentioned it, my main recommendation today if someone wants to get somewhere is to use Ocaml but target LLVM. I'd like to see LLVM re-implemented in Ocaml and parallel developed. Doubt that will happen but using Ocaml to generate LLVM code seems quite doable.

Related: we're wondering how to use Rust because currently all non-Rust dependencies in our org are pulled from:

- corporate source code control systems

- corporate central repositories

- caching/proxying immutable repositories

These ensure all projects are built from known sources. We _know_ we can get consistent builds.

When using Cargo:

- Project owners update projects and don't bump the version. New bugs / security problems could be injected even though we haven't changed a thing internally.

- crates.io isn't always up.

- Trust: we legally can not trust (PCI compliance violations - 2015 rules (viral) ) the public crates.io repository. Besides PCI compliance, it's not possible for crates.io to guarantee perfect security (so many reasons, obviously).

* I'm hoping folks who have addressed this issue (or are addressing it, or are planning on addressing it) would comment.

> Project owners update projects and don't bump the version. New bugs / security problems could be injected even though we haven't changed a thing internally.

You can't update a crate on crates.io without bumping the version. Once a version is published, it cannot be removed. (It can be "yanked," but even yanking it does not make it completely inaccessible.)

> These ensure all projects are built from known sources. We _know_ we can get consistent builds.

Cargo isn't coupled to crates.io. You can run your own registry index. (Note the `[registry]` config section: http://doc.crates.io/config.html) --- All of the code that powers crates.io is open source. On top of that, crate dependencies can be specified via git URLs or locations on disk. Repeatable builds are well supported IMO.

Ahh PCI compliance :)

I think Burntsushi gave a good answer here. We want these features! It's a matter of getting the requirements correct, and then helping build them. We have some of this stuff already, and are working on what Firefox needs, which is very similar, but would love for anyone who has a stake in this to help tell us about what they need, specifically. If that's you and or your org, starting a thread on http://internals.rust-lang.org/ would be quite helpful.

You can use git or path deps if you want; or set up your own registry (there is code for it, but I don't think it's easy to set up yet. IIRC it's planned, thoughts welcome!)

> Project owners update projects and don't bump the version.

You cannot update code in a crates.io dep without bumping the version. And new versions only get pulled in when you do a `cargo update` or you update a package which bumps the version number of its dependency.

Reply on news.ycombinator.com