Hacker News Reader

Comment by tomohawk | Hacker News Reader

One of the biggest benefits of go vs python is that you only manage your dependencies during build. You end up with a single executable that is easy to deploy and mostly immune to the deployment of other things. In python, you have to also worry about deployment. To be safe, you need to carry the vm and other dependencies with you.

This means you can't just give any python program to someone and have it just work, unless it's pretty trivial.

View on HN

Not entirely true anymore.

Nuitka (nuitka.net) has been able to compile python programs to a single stand alone executable reliably, robustly and easily for a while now.

It's a real gem and deserves to be more popular. We are very far from the quirks of pyinstaller or cxfreeze : it just works, even with numpy or qt.

Yes, go is still superior for deployment :

- it can cross compile

- executables are smaller

- it's built in so it's just an easier experience

- it doesn't depend on libc

But it's important to spread the word: the time were your python program was hard to ship is over.

Just use nuitka.

op literally said:

> To be safe, you need to carry the vm and other dependencies with you.

You do the equivalent with Go too, it's just much smaller and the norm, rather than something remarkable.

Go also carries all its dependencies with it; this is why it produces large single static binaries.

Python lacks a tool to painlessly produce the same, that is, a self-contained binary that includes the VM and all the dependencies. Determining Python dependencies may be slightly tricky, because `import` is an executable statement, and dependencies may be native-code libraries.

There are several tools for that, but all have their downsides, and none is in widespread use. Polishing such a tool until it's as simple and fool-proof as Go's build tool could be quite beneficial for Python community.

Everyone seem to agree the generation of a static binary is easy with go. That's not my experience as soon as CGO and the glib is involved. I would gladly pay 100$ to make it work on my open source project using libvips to anyone who can create the static build: https://github.com/mickael-kerjean/nuage

Why does it have to be a self-contained binary? When you pack everything into an virtualenv you get more or less the same result, don't you? If you want to have a single file you can zip it :)

Usability-wise, there's a huge gap between even low-effort and no-effort actions.

Another upside of a single binary is that it's not easy to modify; you can even sign it. Quite an upside if you want to audit what you deploy to your production that handles millions of dollars.

Can't you tar the python venv and sign it as well? Might be more difficult to get reproducible builds tgisbway, but it's sort of the same thing isn't it?

You can deploy an executable on a server just by typing

    GOOS=linux GOARCH=386 go build foo && scp foo user@server:

That's a lot of time saved, especially when you do it on a daily basis, and not something you can do as easily with virtualenv and such.

Yea I just do not buy it. How much time is saved? Is it writing an additional SSH command to extract the file or hitting the up-arrow on your keyboard to repeat the previously run command?

Not a lot, but when your test environment is on a remote server (because it replicates the production DB and you can't have that on your dev machine for security reasons), you tend to do it a lot. Add fast compilation to the mix, and you have a very productive workflow. `make && zip && scp && ssh unzip` is way slower than `go build && scp`

Then you don't have any metadata about what's installed nor a way to roll back. I'd rather have automation that runs "sudo yum update $MYPACKAGE".

Good point. We found ways to overcome these issues, but that's a tradeoff, for sure.

There's also xar[1] (and similar things like par), which provide similar functionality in python in a managed way.

Usage would be

python foo/setup.py bdist_xar && scp foo/dist/foo.xar user@server

if my understanding is correct.

[1]: https://github.com/facebookincubator/xar

Right, its not magic to build such a tool. But until it exists, this is a benefit for Go. This is also why I keep using Tcl, with tclkit you can build single-file executables for you tcl programs.

Pyinstaller solves the problem for me, but it can (in some particular cases) be a real bitch to configure properly with your dependencies at first.

Such a tool now exists and is called nuitka. See my other comment. It's very mature and can be trusted even with compiled extensions.

Speaking from the linux distro perspective, this is not true.

You can develop an application in python that utilizes distro-supplied python libraries. If you package your library in .deb or .rpm, it's trivial to utilize the dependencies from the distro.

Otherwise, you can install from pypi via pip, and especially into a virtualenv. You can declare each dependency's specific version if you so choose. Modern distros support python2.7 and python 3.x.

If you're on some other operating system, yeah, using python sucks for you.

While it's fair to say python is not statically compiled, it's no different than managing dependencies in other dynamically linked software.

Unlike python, go applications have to be cross compiled. For the most part, native python code will run anywhere the interpreter runs.

In the world of containerized applications, vendoring dependencies is a moot point anyway. We're all shipping around big tarballs with everything baked in anyway, so go doesn't offer any advantages in that space other than smaller binary size.

Not to say that go doesn't have some advantages over python, but I think the dependency management problem is vastly overstated, at least when it comes to python.

> You can develop an application in python that utilizes distro-supplied python libraries.

And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs. Also your app is no longer for Linux it's for Ubuntu 18.04 updated roughly on 2018-08-03.

The same goes for the actual python. Distributions apply lots of downstream patches and backport fixes so your python 2.7 is really redhat-python2.7-13 so you might want to test against that -- or you can bundle your own and be done with it.

Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.

> pypi via pip, and especially into a virtualenv

I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?

> And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs.

Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).

> Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.

If you write software that will be run by others (which usually means open source, probably libraries), yes. If you write software that will only be run by you (pretty much all dynamic websites a.k.a. webapps land in this category), you don't want to have three different distributions in half a dozen different versions anyway, so you can pin yourself to the target environment just as well.

> If you write software that will be run by others (which usually means open source, probably libraries), yes. If you write software that will only be run by you (pretty much all dynamic websites a.k.a. webapps land in this category), you don't want to have three different distributions in half a dozen different versions anyway, so you can pin yourself to the target environment just as well.

Yeah, my primary software development mode is 1) gather opensource software, 2) build web app or platform-specific app.

> Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).

It's crazy that this even has to be said in this day and age. I suppose distro specific patches to maintain a secure and stable api has gone away in favor of some vendored static dependencies that may or may not ever be upgraded, and rebasing your dependencies introduces the same set of problems.

I foresee dynamically linked go with platform specific binary packaging becoming the future. Recompiling the same bits of software ad infinitum is probably going to get old.

My industry, vfx, isn't as ubiquitous as web development, but it is fairly large with commercial interests to motivate making the "right" decisions and not too odd of a use-case. I've heard some second-hand stories about Guido crashing on the couch of some guys at ILM back in the "early days" helping to spur adoption. However, we're still pretty solidly on Python2 with proposals to start adopting Python3 next year. Most of the commercial applications run Python inside them and the apps are very, very scriptable.

Houdini is a fairly ubiquitous tool for creating dynamics; fire, water, destruction, dust and has been around since 1996. In general, Houdini users tend to prefer Ubuntu and most medium to large vfx houses use CentOS/RHEL (Pixar, DreamWorks, ILM).

For Linux and macOS, Houdini uses your system's Python but it has to monkey patch parts of the standard library in order to work smoothly. You can set a flag (and on Windows this is the default) where it will use the Python shipped with the application.

At my current job we're using slightly out of date versions of everything, which is quite common in production. We're running CentOS 7.2, Houdini 15.5 (released Nov 2017), and Python 2.7.5. If you call "httplib.HTTPSConnection('google.com')" it throws an exception because of ssl library changes across 2.7. I haven't tried any other versions, but the requirements for the current version of Houdini says "CentOS 6+ (64-bit)" (among other OSes).

This isn't unique to Python, but it's very hard to properly support real-world clients and dynamic libraries among operating systems, commercial applications, and less well funded open source or independent efforts.

My M.O. is if a technology is core to your business, you should control it. Which means not using the OS' Python. Upgrading Python and OS independently makes so many things much easier when you have a large codebase.

> Or use a distribution that does not break shit left and right, like Debian or Red Hat (CentOS).

Sadly, there are a community of devs who want to rely on libflakey 0.0.1-beta, released an hour ago, and think that waiting for APIs to stabilise & distros to securely package things is unprofessional (!).

Using libflakey (lol!) is not necessarily a bad thing; The proper course would be to pull out what you need, make sure the interface you care about is sane and works well; you can revisit the upstream in the future.

Raw npm-style clone-from-master seems to be prolific, though.

> I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?

If the users of your software are end-user desktop types, I can see your sentiment. My software is typically for building software (aka, libraries) or users that know how to compile things.

Unless your application also contains other assets like images etc. Now you suddenly need to package and install/deploy your Go application.

Or you just bundle them all in to the binary like a monster.

Hello Win32 rc files all over again.

I do miss editing those with Resource Hacker or XN Resource Editor. I translated tons of Japanese and Chinese software into English that way back in high school and early college.

Good times.

Is there an accepted way to do this?

I know Consul was using https://github.com/jteeuwen/go-bindata [1] a few years ago.

Personally I'm not a fan but I can totally see the appeal.

1. https://github.com/hashicorp/consul/pull/1543

go-bindata is the conventional method, but I'm not entirely sure which fork is the most maintained one right now.

I wanted to statically compile a single HTML template file recently, and I liked using vfsgen [1] and vfstemplate [2].

I also found this comparison of libraries pretty good: https://tech.townsourced.com/post/embedding-static-files-in-...

[1] https://github.com/shurcooL/vfsgen

[2] https://github.com/shurcooL/httpfs/blob/master/html/vfstempl...

And TBH properly packaging Go program is a nightmare if you want packager to build it and the build machine has no access to net.

Statically linked binaries are hardly new. As with everything they come with tradeoffs.

Depends on your usecase, really. Static linking has been there for a long time and it has its disadvantages. I have more than a dozen programs on my system that share the same Python runtime and libraries. Compare that to every Go program shipping its own copy of the same things.

>every Go program shipping its own copy of the same things.

Is that really an issue? It takes up a bit more space, but nothing that's really any concern on a modern system. People often resort to using Docker to manage dependencies in other programming languages and end up shipping an entire copy of an operating system.

I honestly don't see many downside to shipping a static binary. At least in that case you can chroot the entire thing, something I really wish Python (and most other languages) would make a whole lot easier.

Security fixes require a recompilation, that perhaps the largest downside, but if you use Docker, then you need to rebuild that image as well.

Also, deploying to production is as easy as writing a small shell script to simply copy the new binary and restart your service definition. (or using ansible/puppet/chef/saltstack et al)

Yes, if you can keep things simple. If you've fixed a ux bug and your binary depends on a certain new version of an image/icon etc - it may or may not be that easy.

You can bundle assets, like icon, in your binary. In that case changing an image results in a new build and a new binary. Then deployments are still just one copy via Ansible, or similar.

Can? Sure. Should? Sometimes.

Absolutely true.

You should make it easy as possible for users of your software .

Of course you can do the same with python - that's what docker is for.

Reply on news.ycombinator.com