It’s so thoroughly broken, every day a dev on some team gets their poetry env entangled with some system installed python, or numpy suddenly decides all the CI builds will now compile it from scratch on every build, or.. Today it was segfaults on poetry version X on the M1 Mac’s, that went away in version Y but of course version Y broke pandas for the windows devs..
What is impressive, is how so many in the industry have forgotten about them.
Save in requirements.txt. Includes sub dependencies that are system specific.
Install from requirements.txt on different machine and get errors.
Uninstall dependency and save to requirements.txt.
Look at requirements.txt and see that sub decencies are still there.
What is the right way to avoid these issues on Python?
Or junk all this and just use poetry, which manages both the abstract dependencies (pyproject.toml) and concrete ones (poetry.lock).
However: poetry still falls short in managing the python runtime, I am continuously having to divert time to help our data scientists untangle the messes poetry makes with virtualenvs and their local python setups.
Also, they broke backwards compat on the lock file format? So now devs running newer poetry versions break projects for devs on older versions because the lock files aren’t compatible?!
* use a virtualenv, so that dependencies are installed per project and not globally
* only specify my top-level dependencies in a requirements.in file, and let pip-compile (a dependency, part of pip-tools) compile my requirements.txt
So far it has served me well and I've not encountered any error due to this method.
I cannot claim it is "the right way" though, just something that works for me.
Keep your high-level dependencies in requirements.in.
Create a virtual environment using venv.
Run `pip-compile resolver=backtracking`. It autogenerates a requirements.txt file with all dependencies and sub-dependencies and their versions. This essentially acts as a lock file from other languages/frameworks.
Install from the autogenerated requirements.txt file in the venv virtual environment.
If a dependency changes, change in requirements.in, recompile the .txt file, and reinstall.
This is new for me. I used `pip freeze > requirements.txt`
I think this should be emphasized more, I don't think it's just a matter of preference, not separating packages between projects using virtualenvs will land you in a world of hurt as soon as you want to update or uninstall any of them and they're hopelessly entangled with system packages and other projects.
Maybe you are an accidental accelerationist? Using the system Python is fastest way to a broken system and a broken Python.
Most projects state something like "run pip install whatever", which people will then do. If you're lucky, it will ask for sudo. If you're even luckier, you stop to think before entering your password.
That default Python env means I don't ever modify the system python for any reason. I type `python` and I get my own Python3 from the base.env
If you use homebrew, there are two settings that help keep your old python binaries around so that venvs that reference those old envs don't break.
HOMEBREW_NO_CLEANUP_FORMULAE, this takes a lists of packages to not cleanup
HOMEBREW_NO_INSTALL_CLEANUP, prevents old packages from getting removed
I actually far prefer the Python method to, say, Go's GOPATH. But, I far prefer Rust/Cargo to either.
I wish Python got a standard integrated solution for package management that works.
pip doesn't cut it in my book since it doesn't let you specify dependencies reproducibly. It either doesn't support lockfiles or ONLY supports having a lockfile, without dependencies.
venv also doesn't cut it since you have to remember to explicitly activate it and keep track of which venv is activated right now.
If we take a look at the much-maligned Node.js+npm, it's still far better than what built-in tools in Python let you have. Yes, Node.js doesn't provide full isolation from globally-installed node_modules, but at least it supports a local node_modules directory and lets it take preference. Notably Node.js+npm, with all its warts, doesn't suffer from the the two issues I've mentioned above.
Debian is particularly bad because they'd comingle python 2 and 3 packages in the same filesystem locations if you used pip out of the box.
That was a horrid discovery on my part. I assumed, coming from java where this sort of thing has been a long solved problem, that no one would be daft enough to do something like that.
I always use the system Python plus virtualenv and that works fine, it's certainly never broken my system...
I don't even know by what mechanism that breakage would happen.
If you just run "pip install" doesn't literally tell you to use the --user flag or virtualenv?
I guess it's probably an easy mistake to make.
Once you set the version of python in the virtualenv and activate it, the right python will always be there. The virtualenv even has an instance of python in it's bin. When that python is run, whatever's installed in the virtualenv comes with it.
Poetry, however, does suck even though it's using virtualenv underneath. It's doing frankly too much and seems to have a lot of regressions. In the end it should only be looking at the contents of the virtualenv and using pip to install the correct packages into it.
Edited to add: With the last version of poetry causing issues in my development env, I'm about ready to ban it from our team.
Still, this means you can install multiple system Python versions and your venv will always contain an executable file of the version the venv was last installed/updated with.
╭─android@localhost ~/tmp/py
╰─$ python3 -m venv testvenv
╭─android@localhost ~/tmp/py
╰─$ file testvenv/bin/python3
testvenv/bin/python3: symbolic link to /data/data/com.termux/files/usr/bin/python3So to fix it you would need to change that, but then you would break a load of Python libraries which are most of the reason people use Python in the first place.
Deno doesn't really have that issue so badly because JS imports are ... not completely sane but they're quite a lot saner.
Anyway I think if you were going to break compatibility with Python by fixing the packaging and import system, it's not a much bigger step to fix the rest of Python too and then you have Nim or Lobster.
> What problem are you having with imports?
This is a Python function I wrote some years ago to import a module when given a path:
def module_from_path( name, path ):
### thx to https://stackoverflow.com/a/50395128/7568091 ###
### thx to https://stackoverflow.com/a/67692/7568091 ###
import importlib
import importlib.util
spec = importlib.util.spec_from_file_location( name, path )
module = importlib.util.module_from_spec( spec )
sys.modules[ spec.name ] = module
spec.loader.exec_module( module )
return importlib.import_module( name )
I used to work at a company where they had a custom import system and it was a nightmare.
The dynamic module:
» cat sub/test_module.py
def hello(name):
print(f'Hello {name}')
» cat test_script.py
from importlib import import_module
full_module_path = 'sub.test_module' # no ".py"
module = import_module(full_module_path)
# optional, if you want to use across the project:
import sys
sys.modules[full_module_path] = module
print(sys.modules[full_module_path])
# end optional
# try it:
module.hello('there!')
» python3 test_script.py
<module 'sub.test_module' from '/home/foo/sub/test_module.py'>
Hello there!
https://docs.python.org/3/library/importlib.html#importlib.i...
In general, I don't see the intrinsic complexity being reduced much more than this:
from importlib import import_module
module = import_module('sub.test_module')> In general, I don't see the intrinsic complexity being reduced much more than this
you know in NodeJS I do `const tm = require('sub/test_module');` which I for a number of reasons do find inherently less complex.
https://stackoverflow.com/questions/14132789/relative-import...
And you need (maybe used to?) an __init__.py file in each package.
But once you know that, it's straightforward, no? Certainly not, "only a few people in the world know this..." level. Don't think I'm some kinda level-10 genius here.
Yes because referring to the filesystem is logical and intuitive and referring to some abstract "package" that changes depending on how you invoke Python is insane.
> But once you know that, it's straightforward, no?
I wish!
Close to none of the notation of Python works the same as a shell. Even quotes, which are superficially similar but have significant differences.
> abstract "package"
Packages are not abstract, they are simply a folder with .py files inside. Sounds concrete enough to me.
> changes depending on how you invoke Python
I'm not sure what this refers to? I know you can change PYTHONPATH. And the current folder portion changes with it.
from foo.bar import Baz
And because of how Django's settings work (import django.settings and all your stuff is magically there), people in my company have gotten into the habit of putting code in settings/__init__.py to do the things they need, like retrieve secrets from AWS.
Which I don't want to have happen when I'm trying to import settings.bla_module to use a type in a unit test.
Yes, and it needs to—to find the variables there, which may/not be other packages.
> how Django's settings work
Right. Django is great and popular, and so I think many folks don't realize how downright clunky it is in a number of places. Many parts of it are over-engineered like this one. It has improved over time but some things are set in stone due to history. I'd like their settings redesigned myself.
> __init__.py to do the things they need, like retrieve secrets from AWS
No, no, no, no... NO! ;-)
Imports should only create variables, define functions, and/or alias a package. They should never, ever run lengthy computations or access the network. That's basic software design 101, unfortunately not taught everywhere.
I don't see this one as a clear flaw of Python. "Dumb" coding can happen anywhere. I suppose it could prohibit this kind of stuff, but that might prohibit other cool things that might be reasonable in limited situations. It is a "consenting adults" design, with those trade offs.
Fast forward 30 years and we really don't use most systems like that anymore, especially in production environments. We run things in VMs or containers and build them up from scratch with ease--it's all just cattle. Python hasn't really adapted to that new reality.
You might think this is silly but think about something like making a systemd service to run your python script in a venv--how do you do it? You have to activate the script or call its python bin, but it's not obvious how to do that in any python docs.
The Python docs do cover this topic, and do so rather well: https://packaging.python.org/en/latest/guides/installing-usi...
How do I ensure that at each customer site, my library has the set of dependencies it needs, with versions that it is known to work with? Each installation method has a different constraint solver, a different means of specifying dependencies.
If your question is "how can I ensure technical end-users have the same set of python packages as I do for the code they run using standard pip+venv?", the answer is to pin dependencies in a requirements.txt file.
If your question is "how do I stop end users from installing dependencies for my software cowboy-style?", the answer is to write installation and usage instructions, and/or include an AIO run script.
If your question is "how do I package my library so that end users' package managers know my library's downstream dependencies when they install it?", you build a wheel using `pip wheel`, which again relies on a requirements.txt. If I'm understanding you right, you're mistaken that you have to handle the package managers separately; they all use pip + wheels under the hood. Conda is a bit of an asterisk in that you can package things differently if you desire, but it plays nice with pip + wheel builds too.
https://docs.conda.io/projects/conda-build/en/latest/user-gu...
It doesn't matter which language you're using, any non-trivial code you have is going to have dependencies which you will need to pull in.
If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!" problems that used to plague people.
You disagree that "you shouldn't need a clean OS every time you want to use a new language version"!? ... I don't even know what to say, that's not something you can disagree with
> If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!"
This is what testing is for. You don't need to cripple your development environment to have confidence your code works in other environments. Talk about the ends not justifying the means.
On the one hand, it bothers me that software is so brittle that you need a container around it so that everything is just right. On the other hand, you can argue that containerisation is a clever way to make it all more robust.
If you put everything into an Uber Container, the same problem would surface.
Containers exist to solve the fragile dependency/dynamic linking problem.
I built this resistance to docker about 5 years ago, is it (or was it) still correct? Is there a best practice for sharing GPUs with containers that's not a pain in the ass?
Edit: I should specify that I typically develop on Ubuntu, but occasionally will do so on Windows (without WSL).
Python doesn't need "something like deno". All programming languages need package management.
> And it works well on Windows. No idea about Mac and don't care.
Package management is a cross platform problem. You may not need it but that doesn’t make the current solution good.
This is where I kinda regret switching from and miss chef. Pick up an RPM/APT package from vendor, shove it into your package manager, the environment works. It developed a sufficient amount of weird behaviors later on, but that's besides the point.
Instead we have a really precise documentation of 4-5 steps to get everything installed in a way that should overall work, and each step is annotated with 4-5 ways that look deceptively good but end up being even more quirky. And when you do all of that, and review it twice, it still doesn't even work consistently across 6 workstations. 3 work the same, 2 have interpreter discovery anomalies, 1 has import conflicts with seemingly unrelated other installed stuff.
Whenever I have a working python VM / container build or installation, I kinda feel a need to shut up about my jenga tower because otherwise the universe will find a way specifically to ruin my day.
I hate how I have to feel like this about a language I really like.
And then
Python3 -m venv envdir
source envdir/bin/activate
bin/python3 -m pip install --upgrade pip
And similarly the rest of the packages needed for the project
This does not mess up the system library or other python projects
This step is the least intuitive until you've done it enough times that it's muscle memory.
Can it be better--yes. Is it worse than _any_ other language's packaging if it were to scale to the number/kind of cases Python supports/used--no.
"thoroughly broken" is useless. Are there bugs--most definitely yes. Do report them, or even help fixing them if you care.
There are many many different use-cases where different approaches are preferable (no size fits all).
Personally, most most of the time packaging is a non-issue e.g., pinning versions and cache are highly recommended for CI to avoid unnecessary surprises, flakiness.
- the guy who created C++
I always bring up Go in such arguments - according to the survey [0] done by the Go team, 93% of Go users are satisfied with the language.
In Python land one of those seems to come along every couple of years. They start over and implement a solution for some subset of the problem space. Then they realize the Python packaging mess is much, much bigger than anticipated and progress stalls. At that point there are 15 partial "standard" packaging solutions for Python, where there were 14 before. None of them are feature complete, and they don't really coexist well or at all.
Cue the obligatory https://xkcd.com/927/ and https://xkcd.com/1987/
From that presepective , most python libs are written in python only.
If you get any python job, you'd get a lot of side-eyes for not knowing of/how to use and initialize basic venv package sets. Even if the shop uses conda.
People were not impressed.
Hence the previous article (https://bitecode.substack.com/p/relieving-your-python-packag...) advocating to go back to the basics yourself.
See the reddit thread to witness exactly this in action:
https://www.reddit.com/r/Python/comments/124hktv/comment/je5...
Now we have a never ending train of hotshot developers that think they can "solve" Python packaging with another tool, without realizing that you can't design a tool to fix a broken 15-year-old system. It's not just a "there are now X competing standards" issue, it's also the fact that anyone who makes an earnest attempt is doomed. They will never be able to get it past the point of universally good enough.
I lament the fact that I'm dragged back into the Python ecosystem every time I need to do something with a PyTorch-based tool someone spent years of effort building. I would never recommend the language to a newcomer. Ruby pollutes the global namespace, and Gemfiles execute arbitrary code, but at least with Ruby there's Gem, and then there's Bundler, and nobody complains.
Will we forever be beholden to the mistaken decisions of the original developers in the early 90's, just because Python is simply too widespread? Because even people like me who detest the packaging experience are forced to go back there because nobody's going to rewrite the hottest ML project in Elixir or something? Even the simple mistakes like putting venv files in Scripts/ instead of bin/ forcing one to work around it in cross-platform CI make no sense in hindsight. Nobody remembers why it was made that way after so long. Now it just floats over you like a spectre.
Python's environment management is just the tip of the ice berg
Dynamic typing
Gil
Speed is slow at best
Async streams having its set of non tractable issues
Debugging by printing pdb all over and breaking