Hacker News Reader

Comment by draw_down | Hacker News Reader

I wish!

Could you elaborate on the issues you have with the current system?

Deploying a python application is still a horrible experience. My best attempts so far involve doing `pip wheel` during the build, then discarding anything that's platform dependent and fetching their source dists instead, then bundling that together with a script that sets up a virtualenv and installs all these dependencies and the project being installed ...

Compare to a jvm uberjar.

Yep, I'm doing the same thing to deploy Python apps on Amazon's Elastic Beanstalk... I write an "abstract" app-requirements.txt (which typically just has a single line for my application's own package), create a fresh virtualenv, `pip wheel -r app-requirements.txt` to resolve dependencies and build wheels for all the packages, and finally `pip freeze > pinned-requirements.txt` to create a list of version-pinned packages.

Zip it all up and then deploy with `pip install --no-index --find-links=/path/to/wheels -r pinned-requirements.txt`. Et voila! Binary artifacts that don't suddenly break because PyPI goes down or a package maintainer uploads a new version that unexpectedly deprecates an API that you're using.

You might want to consider checking out Pex (https://pex.readthedocs.io/en/stable/). It helps you create a single, executable artifact for your Python project very similar to an UberJAR.

Except Python you can often run from git. Git pull do some pip done.

You can do the same things in Java land as long as you have used a build system written in the last decade such as Maven or Gradle, you can just as easily pull down a git repo and run mvn package

Well that is

a) A 30 sec - 10 minute process to build (vs 0 in python)

b) Weird because Java has a fatjar concept.

Python and Java are different. The easy path to deploy them doesn't need to be the same.

You may need to make them the same for ops sanity.. but if you are just doing 1 or the other, meh

> a) A 30 sec - 10 minute process to build (vs 0 in python)

Have you ever run someone else's python code? You have the exact same issue pulling down their requirements.txt, which is often not enough because you need to apt-get other libraries, try pip again, apt-get more, try pip again, etc.

> b) Weird because Java has a fatjar concept.

Sure, that's what you would do if you wanted to deploy an application to proc, my point was that it supports the same git checkout workflow if you want it to, which can be useful when dealing with other people's code and you want to make changes and not just run whatever jar they made.

To be fair, you need a python runtime installed to run python, the same way you need a java runtime installed to run uberjars.

...but, the deployment is a single artifact.

I don't think doing a git clone, followed by a pip install and waiting for all the dependencies to download... and compile... (or not, if you have the wrong system packages installed) really compares.

Why not?

If you are doing a blue/green type deploy.. you are basically building up a new VM to be your new environment. It can do all kinds of download and compile type stuff. Who cares if some is pip installing?

Having to explain the concept of virtualenv is still a pain compared to local install à la node.

"How do I make an exe" with my PyQT app is still a very hard question to answer.

Web deployment is still hard in Python because of lack of general hosting support if you don't own your server.

The situation is way better, and I'm very glad of the work done, but it has still a long way to go.

> Web deployment is still hard in Python because of lack of general hosting support if you don't own your server.

Does conventional shared hosting actually matter anymore, for application developers as opposed to someone just running a WordPress site? Virtual private servers are just as cheap now.

Yes it does, because a lot of people needing to deploy tools are not dev.

> Web deployment is still hard in Python because of lack of general hosting support if you don't own your server.

GAE? PythonAnywhere? Heroku? OpenShift? Docker-based stuff? https://wiki.python.org/moin/SpecializedCommercialHosts maybe? https://wiki.python.org/moin/OtherCommercialHosts if that wasn't enough?

0.1% if the commercial offer for PHP, and they all involve using the command line or know a bit about coding. Again, it's better, but many people I know run PHP business with zero coding knowledge. They upload to ftp scripts they bought on the web, and make a living out of it. Some are kids trying stuff. The little things matter.

It's a good summary of the context, history and tools that have been cleaned up but sweeps an awful lot under the rug with "Then, for each project you want to do, make a new virtualenv [...]". What's that and why are you doing it? Oh, because the packaging doesn't work.

The packaging works fine. Do all Java apps running on a host use the same classpath pointing to /usr/lib directories full of JAR files? Of course not. That's all that virtualenv is doing; the equivalent of managing /lib directories for an invocation of the interpreter.

I don't follow how these are at all similar other than you saying they are 'equivalent'. Typical java apps get their dependencies from the same maven-managed local repo. The tool itself is baroque but conceptually this is much simpler than creating magical 'environments' - the repository of dependencies, what dependencies a particular app uses and what version of the runtime it's invoked with are all separate and orthogonal to each other.

An even simpler way to think about it - imagine you just invented python and are designing the tooling around dependency management and packaging. Would you come up with 'environments'?

> Typical java apps get their dependencies from the same maven-managed local repo.

What? No, they don't. Typical Java apps are deployed on systems that don't even have Maven installed. Typical Java apps are executed with a classpath argument pointing to the libraries installed for the application. What are the directories containing those JARs (or an uberjar) except an "environment?"

Hell, the Java classloader interacts with the classpath in pretty much the exact same way that the Python interpreter interacts with a virtualenv.

Is there anything that even remotely comes close to bundler in power and ease of use? Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done? Or suppose I have different environments a project can be run in that should load slightly different set of dependencies?

Bundlers default-global behavior is a disaster. Last time I got hired at a ruby shop none of the scripts worked, most projects didn't have a lock file, etc. Because they just ran the things directly, using whatever from the global environment.

If the gems are installed globally and no Gemfile.lock file is used, of what use is bundler anyway?

While bundler certainly have warts, the disaster in that case is not using a lockfile. That completely defeat the purpose of the tool, to get repeatable installations.

> Is there anything that even remotely comes close to bundler in power and ease of use?

Maven and co.

> Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done?

Change the version number in pom.xml/build.sbt/build.gradle. Done.

> Or suppose I have different environments a project can be run in that should load slightly different set of dependencies?

Define a new project that depends on your main project and the extra dependencies.

Ah yes. The question was about the Python ecosystem. There are other good solutions for other platforms.

> Is there anything that even remotely comes close to bundler in power and ease of use?

Yes, pip, since before bundler existed.

> Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done?

pip install --upgrade package

> suppose I have different environments a project can be run in that should load slightly different set of dependencies?

venv activate myproject

There's a bit that's still lacking in Pip. For instance, Pip only installs new packages into your venv, rather than trying to keep it synchronized.

I've run into bugs where someone ripped a library out of requirements.txt because they thought it wasn't being used any longer. Turns out it was still being used, but the dev couldn't tell because the lib was still in their venv. Tools like Bundler will actually take the dependency away from you.

Python's packaging system is great compared to C, but it's primitive compared to Ruby, Haskell, Clojure, and lots of others.

https://github.com/nvie/pip-tools solves this by providing pip-sync, which will uninstall removed packages.

pip doesn't even use a solver to determine which packages to install, it just uses the first mention it finds.

Reply on news.ycombinator.com