Comment by rlpb | Hacker News Reader

Comment by rlpb | Hacker News Reader

Parent Full thread

As a non-Rubyist, I share your pain of having to use Rubygems (and pypi, and CPAN, and all the others). However:

"Since Go is a compiled language, binaries of your app can be precompiled for each platform that you wish to distribute for."

For me, this is worse. There's no audit trail. No way for me to verify that the binary of your app is the same as the source you provide. So there's no way I'm running your binary on my system. I simply cannot trust it with my user account.

Not only is this bad from a security perspective, it's also bad for maintenance. Over time, many end up in a situation where others cannot rebuild binaries from source at all, since binary distribution becomes the norm and the source->binary mechanism isn't maintained except on the developer's own system. Look at the Java community for an example of this. This is a problem because it means that others cannot easy be a member of the community working on the source.

If you want to promote a community around the source (like Github does), you must also provide an easy way to rebuild binaries from the source.

Unfortunately, there's no good answer here.

Distributions like Debian have solved this problem - there are source packages, and there's a standard way of building them to get the binary packages. But there's quite a bit of arcane crud that's built up over the years, making it difficult to learn and understand Debian packaging. And the solution isn't cross platform.

Each language community has produced their own solution (Rubygems, Python eggs, CPAN, etc) but none of them work with the packaging provided by any other language. To me, this may be cross platform, but not being cross language is just as big a problem.

>For me, this is worse. There's no audit trail. No way for me to verify that the binary of your app is the same as the source you provide. So there's no way I'm running your binary on my system. I simply cannot trust it with my user account.

It's not for you then.

For one, billions of people run tons of binary third party apps all day long.

Second, when you deploy to people, they have to trust you. Whether they trust your code is irrelevant. Most of them are non-programmers anyway and wouldn't even check if you gave them a source code printout.

Nothing is stopping you from building the app from source instead, just like every other open source project. But at least with Go you have the _option_ of just downloading and running a binary if you trust the author.

This is true. I do have that option.

But I very rarely trust the author. Most of the time I've never heard of the author. This is a good thing, and exactly why Github has been so successful. By nature, if there's something obscure that I need doing, and I find some software or software module that lets me do it, then chances are I won't know the author.

This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.

> This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.

A rare use case for you, but definitely not an obscure one in general. Besides, trusting the author, in this case, only matters if you're willing to audit the source to verify that nothing malicious is occurring -- something I'm not going to do in general. I mean, realistically, when was the last time you audited your network drivers? your OS? your browser?

You don't have to verify the source yourself. Others can do it. I trust the Debian maintainer for my package. He's a real world person who has had his key signed by other Debian developers.

The fact that the source can be verified also reduces the risk of malicious modification.

It's not necessarily just the author who might be malicious, either. The author's computer where he runs his compiler might be compromised.

Do you trust your compiler?

"You can't trust code that you did not totally create yourself." - Ken Thompson[1].

[1] "Reflections on Trusting Trust", Communication of the ACM, Vol. 27, No. 8, August 1984, pp. 761-763, http://cm.bell-labs.com/who/ken/trust.html

Could you expand on what you mean by "I very rarely trust the author"? Isn't using their work a statement of trust?

Source or binary, you are still embracing their description of tasks for an automated machine to perform very rapidly.

If you are wary of what they have done, it's true that source is somewhat more accessible to examine than machine code, but for any non-trivial program source is still quite expensive to evaluate.

I don't necessarily need to trust the author directly. If he provides source, and the source is known to be the true source of the binary I'm using, then the source can be checked by others, even after a problem has been detected.

> Isn't using their work a statement of trust?

Trust isn't absolute. I might trust an author's intentions, but not that his computer has not been compromised, for example.

Being forced to provide source could be argued to keep authors honest, too.

> This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.

That's very much not true, especially when you start dealing with corporate/enterprise systems (and remember, Go is intended as a systems programming language).

Just last week I heard Eliot Horowitz (CTO of 10Gen) talking about how it's advantageous for them to use Go for parts of their product, because they can provide guarantees about the software that they're running ("Hey, you're running the exact same file as what we're providing you, not fetching dependencies separately and interpreting things on-the-fly, etc.")

From a security perspective, these corporate clients have already entered into contractual agreements where they don't really care to verify that the source and binary refer to the same code.

If I'm paying a third-party company to provide me with software and services, I'm trusting them to provide security at multiple levels. I'm probably not even auditing their source code myself, so the idea that they would ship us a binary that didn't match the source code is something that wouldn't even enter my mind.

What I've described above is not an obscure use case.

Yes, there are other things that could go wrong that statically compiled binaries can't solve, but from the point of view of a bank or a large, non-tech corporation, it's one less link that could fail in the process.

OK, I'll give you that. In an enterprise environment, a company can choose to trust specific vendors' binaries, and this often happens.

However, I don't think this has much bearing on the original pain point. I'm familiar with these sorts of production environments. Running a local store of gems is standard practice already. There is little marginal cost to doing this.

Hacker News hosted the following discussion regarding problems that have already occurred maintaining Go software due to reliance on binary distribution: "Golang dependency hell in Haunts The Manse game" https://news.ycombinator.com/item?id=5796597

In this specific instance it was more of an incorrect understanding of Go dependencies and misuse/lack of source control, but it seemed to at least come close to an echo of the issues you are raising. The specific quote from the comments: "There is no version control for dependencies".

That wasn't a binary distribution problem or anything like one, was it? That was as I remember it a development team that simply hadn't created a known-good repository of their third-party dependencies. While I grant you that's a common pitfall in the rubygem era, it's hard to feel much sympathy as someone who came up building commercial C code, where not having a one-command third-party build-world system was a cardinal sin.

Create a known-good repository of third-party dependencies is so easy in Go. Change the $GOPATH, `go get` all the dependencies and add that folder to version control.

I think I agree. Also, for all the complaints about "magic" in Rails, there's nothing more magical in both Rubyland and Pythonistan than package management, and the explicitness of Golang's "package" system is a welcome contrast.

> there's nothing more magical in both Rubyland and Pythonistan than package management,

Could you expand on this a bit? I'm interested in your opinions, as it's hard to see from the inside. In Ruby, it's just things modifying the $LOAD_PATH, which should be basically the same as changing your $GOPATH, no?

I don't know much / anything about ruby package management, but if it's commonplace for ruby packages to change the LOAD_PATH (which I guess is similar to PYTHON_PATH / sys.path in python ?), that is really quite magical (by which I mean, causes unexpected behavior).

What conroy is talking about is creating a static set of dependencies that you check into your own source control, because go doesn't let you specify e.g. github commit hashes to "pin" a package dependency to a certain revision.

So a workaround for that is to (as a one-off) change your GOPATH ($GOPATH/src and $GOPATH/bin is where the standard "go get" / "go install" etc put things) to something temporary, "go get" your dependences (which will fetch them from github or wherever at the HEAD revision). Now run your test suite to make sure everything passes. If it does, great, take the current $GOPATH/src and check the contents of that into your own revision control. Et voila, ghetto-"pin"ning.

The packages themselves don't change the LOAD_PATH. rubygems/bundler do. Both explicitly add the paths given in the gemspec (lib/ by default). I wouldn't call that "magical".

Yes, you would do the exact same thing: set your $GEM_HOME and gem install, or use 'bundle install --path.' It'd be basically the same thing.

Reply on news.ycombinator.com