Reproducing Go binaries byte by byte
blog.filippo.io
blog.filippo.io
This seems worrisome for the privacy of builders. What if someone wishing to stay anonymous built and distributed a Go binary from a system where GOROOT included their home directory and revealed their identity? Am I misinterpreting something?
More about GOROOT here[1].
If you install Go into your home directory, then yes that path would show up in resultant binaries.
[1]: https://dave.cheney.net/2013/06/14/you-dont-need-to-set-goro...
In the meantime, if you want to be anonymous, build in a non-specific path like /build. And perhaps more generally, you might want to use a generic username like "user".
Even if the goroot is not in a suspicious location, the actual location of each source file on disk is compiled in as well (which almost certainly will be in your home directory).
Copying it into `/usr/app/src/$pkgPath` in a docker container makes the path generic and also ensures various other details of your environment are less likely to leak out due to the mount namespacing.
Alternately, building it on a CI system like travis and ensuring those are the only binaries you ever distribute is marginally safer.
Modern environments are to complex to manage safely with whitelists. Someone will always forget something.
Honestly this thread is going in the same direction most other threads I see about any flaw in GO seem to take. Users popup to defend GO and try to downplay the issue or shift blame towards something else, like you not framing the issue in the "GO way" (I guess not wanting to expose your username in release binaries is just not the "GO way", if you're not using Docker for release builds your username showing up is your own fault.).
This is also a non-issue because if you're compiling binaries for redistribution you'd disable the debugging references (via compiler flags) and be building your source in a separate environment to your development environment (eg using Jenkins). Heck, even just a docker container on the dev machine would work.
So to recap:
1) not a problem with Go; a problem with debugging references in all AOT compiled languages.
2) not a new issue; I've been observing this in other languages for at least 2 decades.
3) it's easily fixed by implementing any or all of the follow solutions:
i) build automation / CI pipeline (your build environment becomes separate from your dev environment)
ii) docker or other build container or VM (same reason as above)
iii) just disabling the debug references at compile time
> I think this approach sounds to be the cleanest. Using -ldflags -s is probably untenable for releases since people expect all of the normal reflection and debugging to work properly. GOROOT_FINAL gets around the privacy issue without making any other major changes.
I'm no stranger to debugging symbols having personal info, that's why I specifically mention `release` builds no less than 3 times. They're even admitting it's a privacy issue but mentioning a fix that's non-default option, but I'll give them that having a fix period is good.
And to this: > The reason people are defending Go is because this isn't a new phenomena in AOT compiled languages
I'd correct that to the reason you're defending Go in that comment, plenty of people clearly don't think it's an issue not because they're used to debugging symbols having personal info, but because of what boils down to "What's the big deal you're using Docker right?".
All of that aside, my comment about my experiences with Go on HN go well past this single thread...
Honestly I see just as many people being highly critical of Go as well. Go is like the Apple of programming languages: it's opinionated, arguably spare in design and often polarises people opinions.
But with regards to your main point, i was talking about production builds as well. I've lost count of the number of times I've pulled build data from a Windows PE or Linux / FreeBSD ELF. I went to college in the 90s and it was a hobby of mine to reverse engineer the build environment of Windows executables (a bit like in this article) because I wanted to know what technologies the professionals were using.
If you RELY on privacy you should have better opsec than just using your daily driver to build your go binaries. Compartmentalizing your life is important if you NEED the privacy.
I'm attempting to say this without sounding facetious: it's generally bad for privacy if you install tools and compilers to a personally-identifiable path, compile programs from personally-identifiable directories, perhaps using personally-identifiable accounts. Using generic paths (which is in this case the Go default), generically-named work directories, and generic-sounding user accounts is a viable mitigation, that has been used by many programmers who get burned by this from various environments.
As you suggest, perhaps there's an education gap here, because this is well-known by security professionals and intelligence services (like the CIA [3]), and less well-known by developers. Furthermore, most of these privacy-leaking behaviors are being (re-)discovered or discussed as a part of the 'reproducible builds' movement (like this one) to attempt to identify context-specific behavior, rather than a concentrated approach about privacy in particular.
[1] https://groups.google.com/forum/#!topic/golang-nuts/oVDD8oPv...
[2] https://groups.google.com/d/msg/golang-nuts/oVDD8oPvDIY/fQ_r...
The first time I saw a go stack trace I strings-d the binary to confirm, as I suspected, that they embedded this information in the binary. I filed it away and didn't think about it much since then (I'm not saying nobody could care, I just don't do things that require that kind of privacy)
Thankfully this is less of an issue these days due to the proliferation of containerisation technologies and automated build pipelines, meaning the build system can be completely separated from developers working environment (eg a Jenkins box)
Repeatr is for -- like it says on the tin -- repeating things; and hopefully, reproducing them. It's all the containment (runc, underneath), plus the ability to pull in various filesystems, specified by hash.
For concrete examples, you can see Repeatr building Repeatr reproducibly here [2], for its own releases (it's a Go project, so this looks a lot like what's going on in Fillipo's blog here). Another example is this formula for reproducible builds of Runc [3] -- it uses cgo, and became reproducible with go1.7, which was very exciting!
The big gain is Repeatr's syntax for multiple inputs and built-in hash verification means you can assemble reproducible environments like this out of multiple filesystems (say, busybox from one place, golang compiler separately), they all cache separately, they all download in parallel, you never have to write your own "tar -xcvf" boilerplate... and most importantly, you get built-in sanity-checking that the upstream tarballs you're using haven't changed either. No centralized "registry" server required, either. Give it a try!
Disclaimer: author, of course :)
---
[1] http://repeatr.io , https://github.com/polydawn/repeatr
[2] https://github.com/polydawn/repeatr/blob/master/meta/release...
[3] https://github.com/polydawn/formulary/blob/master/formulas/r...
Getting the compiler out of the chain would be even better, but trusting trust is a hard problem.
Repeatr formulas are a precise identification of one build. That doesn't mean you shouldn't make a build that's even more portable. It means you can use more than one formula to precisely describe, and test, how portable and flexible your build is. You can also use one formula to describe one step of a build (say, compiling a base image), and then output the resulting filesystem... it's assigned a hash, and you can use this as an input to another formula. Thus, you can use binary images, AND have a complete (recursive!) reproducibility path.
You can use reppl [1] for creating pipelines like this (though it's not yet geared for public re-sharing and comparing of results -- that's a big todo, if you want to lend a hand!).
> trusting trust is a hard problem.
Imagine you had two formulas, both with fully hash-pinned input filesystems, which both build GCC. One uses a GCC binary as an input. The other uses a clang binary as an input instead. When you run each of them, you get a hash of the result filesystem. The remaining step to address Trusting Trust is left as an exercise to the reader :)
---
I do this when I build Go for Solaris:
https://github.com/oracle/solaris-userland/blob/master/compo...
https://github.com/golang/go/issues/13616
See also:
Interestingly, the build host architecture does not matter.
In other words, builds are reproducible across cross-
compiling.
Anyone who cares enough about this must be sure to use a nondescript username / path for creating binaries.
Sniffing around for paths and filenames (and other compiler artifacts, like date/date formats/time zone/time of build, version numbers, locale/language markers, debug symbols, UUIDs, copyright strings, format strings) is an extremely well-known technique for investigating the origin of a suspect binary -- anyone who has ever heard of "malware attribution" is sure to know about this.