Classifying Python virtual environment workflows
snarky.ca
snarky.ca
mkdir newproject && cd newproject
echo 3.11.0 > .python-version # (If different python version)
python -m venv .venv # Make the venv
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
# Custom key-bindings
python_venv() {
MYVENV=./.venv
# when you cd into a folder that contains $MYVENV
[[ -d $MYVENV ]] && [[ ! $VIRTUAL_ENV ]] && source $MYVENV/bin/activate
# when you cd into a folder that doesn't
[[ ! -d $MYVENV ]] && [[ $VIRTUAL_ENV ]] && deactivate
}
autoload -U add-zsh-hook
add-zsh-hook chpwd python_venv
python_venvIs this the behaviour you want, for subdirectories of your project root? Other tools (e.g. git, but also version managers like asdf) tend to look in the current directory, and then recursively try the parent directory to find what the current configuration should be.
For instance, numerical packages (scipy, torch, etc) can add up if they’re duplicated across many projects, and, as I understand it, virtualenv doesn’t do any caching/symlinking.
Well, one of us is incorrect then. I had been thinking for years that one of the key benefits was that virtualenv would not do a blind copy of files (except on Windows...sigh). The `venv` command even has a `symlinks` vs `copies` option.
`python3 -m venv --help`
You then have one python version in the container, and no virtualenvs at all. All libraries get installed right into the main python libraries folder.
I've been using this approach for 5+ years now, and I've yet to hit issues. I read people lamenting issues with python's packaging infrastructure, and with the approach I've never run into any of the issues they talk about.
Pip complains a lot about being told to install stuff globally, but from what I can tell as long as you are careful enough to never use distro-sourced python packages (they're almost always out of date anyways, at least on ubuntu), it's never caused problems for me.
For web-service type things, this has a lot of knock on effects that are beneficial. Each project winds up with it's own IP, if you need to restart things you can often just reboot the VM/container (assuming you have init scripts), process/library isolation issues are effectively impossible, etc...
but I'm still wishing for a native container support from pipenv/pyenv/poetry etc.
I'm 15 years in with python and scientific work. For a lot of years I liked conda but then it got crazy slow. Next I started making conda environments and installing packages with pip. Now I'm experimenting with mamba ("fast conda") and that's pretty good.
Conda envs mean I can experiment with different versions of Python (I'm a co author for O'Reilly's High Performance Python so eg 3.11 and 3.12 are pretty interesting right now). Conda "should" also make identical teaching environments (I teach my own courses). Pip was a pragmatic choice to get installations in minutes not hours in the years when conda was silly-slow.
The above is also all for short -lived research work (my typical client mode for scientific work), so it is probably different to anyone doing long-run dev work, production deploys, or for those not needing non-Python binary support (eg GPU/C/Fortan lib support).
+1 for this. I really like Conda because I use a lot of packages that cannot be installed with only pip (E.g. GDAL). Conda would sometimes take 10 minutes to make an environment.
Mamba has been an absolute game changer. It has rewritten parts of Conda in C++ and added multithreading, so every part of the process - the solving, download, and the extraction - is all Lightning quick in comparison. A 10 minute process with Conda can now take 2.5 mins with Mamba. It also helps that it is also prettier to look at!
- on Windows, using Christoph Gohlke’s wheels,
- on Linux, installing GDAL with the OS’ package manager, then the Python bindings by fixing the version to $(gdal-config —-version) and setting C_INCLUDE_DIR and CPP_INCLUDE_DIR.
It’s a bit more involved, but the speed of pip and conda’s multiple channels and dependency resolution issues make this worth it.
It's probably easier for your students if you provide containers for them, then everyone has an identical environment.
In part it is because Python has modules, not libraries. Modules can have executables bundled with them.
In part, it’s because Python modules packages don’t exactly care about backwards compatibility. By and large, people who put out packages will keep a changelog but not keep the same API going for years. So slightly different version of the same package can give you different results.
Lastly it is because Python does not have a clear definition of a project. If at the language level I could say “this is the root of my project. Do not use any OS-level libraries. Here are my library dependencies and they are to be stores in $PROJECT_ROOT/libs/“ then none of this would be required. I have rarely found that the version of Python itself was a big problem for me, but having the ability for multiple versions to be installed at the OS level combined with the above mechanism would entirely eliminate the need for virtual envs.
People complain about node_modules, but the benefit is that every Node project on a system is isolated automatically and can re-download all of its dependencies trivially (after cleaning them, or after being checked out onto a new machine)
A Node project's system-wide dependencies are:
1. A new-enough Node installation, and
...that's it
The same is true for Go and Rust and other modern languages. Python is the odd one out.
Why are there so many competing tools? Why can’t the community just pick one and make it a good tool for all use-cases? There’s an organization called “Python Packaging Authority”, and it maintains almost all tools mentioned in the previous paragraph (except the stdlib venv, and poetry). If they’re an authority, they should just say “___ is the way to go, we’ll add the missing use-cases to it, here’s a tool to migrate everything else to ___”. Instead, they have done things that support the proliferation of tools, like the PEP 517 (which is a standard API for package managers to talk to build tools).
Another deficiency in the Python package management system is the fact that you need virtual environments to get things done, and that they are often finicky. Some of the modern tools hide the venv from you, but this gets less practical if you’re trying to use things from within scripts, or trying to point your WSGI server at the venv, or in multi-user scenarios. Some of the modern tools put it in .venv and manage it for you, but venvs can randomly break in cases the tool might not be aware of (eg. in some system package managers, upgrading Python to a new minor version would cause venvs to break, because symlink targets moved.)
Node solves this by having a `node_modules` directory. .NET does stuff at build-time that most people don’t need to care about, and then just puts the .dll files for your dependencies next to the .dll with your thing, and the loader looks in the directory your code came from. Python has a proposal for `__pypackages__` [0], but it’s been there since May 2018 with not much progress.
One, Python and its packaging story is old (remember, Python predates Linux). That has given folks plenty of time to either come up with their own solutions since Python predates widespread internet usage (PyPI has not always been around).
Two, a lack of standards. Tying back into the "old" point, not everything was initially designed. There has been work to chip away at this and get more standards behind things, but getting folks to update their packaging code is *hard*; most people copy their packaging code from their last project and don't really try to update it. Getting changes to propagate through an ecosystem as large as Python's takes years; a decade is the typical time frame considered for complete uptake.
Three, the lack of standards means tools come up with their own solution which then isn't compatible with anyone else. That means when someone innovates, it can very quickly get locked up behind a single tool. That means folks end up reinventing the wheel for various reasons (e.g. lock files). Different approaches leads to different opinions, which leads to folks choosing different tools for different reasons. And when that happens, there ends up being a lack of consensus.
And four, this is almost entirely driven by volunteers with very few people paid in any way to work on this stuff (I think there might be like 2 people who contribute to pip, 2.5 folks for PyPI).
And good luck telling someone that their preferred workflow isn't the "chosen" workflow that the whole community is going to switch to. People don't like being told they are going to have to change, especially if they believe their approach is superior for whatever reason. Multiply that by the size of the Python community and you can see the pitchforks quite clearly on the horizon.
Now that isn't to say work isn't being done to improve the situation. The PyPA side of packaging has been working on standards for quite some time and is getting some traction with them (e.g. pyproject.toml is a great example of that). But we do have some more standards to work out. We are also regularly discussing how to come up with some singular tool/UX that people can get behind for most use cases, but see the above comments about the size of a challenge that it is and thus why it hasn't happened yet. But people are aware and trying to figure all of this stuff out.
Which is…not at all justification, certainly.
I havent tried it but my understanding is poetry is a bit more like npm.
Coming from decades spent in C/C++ ecosystem, this is definitely not true. I would say Python users have it rather easy. A lot of Python packaging problems stems from dependencies on messy C/C++ ecosystem.
I lean on Makefile for automation wherever possible, even if it is just a centralized trigger command to run a packaged script (eg scripts/dothething.py). One of my sticking points is how to ensure the Python virtual environment is installed and up to date before triggering any scripts. That is, I clone a new project, cd into it, and want to run some package command inside the expected environment (eg `poetry run scripts/dothething.py`). This will fail if the environment has not previously installed all of the defined dependencies. My annoyance is that requiring `poetry install` to run prior to every command is quite slow, and rarely has anything to do.
Does anyone have a suggestion as to how to ensure the right thing is always done without the slow Poetry resolution? My dream is that in all of my projects you could, say `make test` and Python would optionally create the environment, install the dependencies, and run pytest in the fresh case. The typical development case would confirm that the environment exists and just run pytest.
2) I think that has some brittleness of detecting where the environment lives (as per the author's post). For my use case, I would be fine assuming the defaults, but it is something else to go wrong
Edit: Thinking out loud, I guess the quickest solution is to checksum the pyproject.toml file and store that somewhere. If current hash matches previous, do not run `poetry install`. While that would get me there, I am hoping for a first-party, standard solution.
This is the kind of stuff bazel/ninja do.
Make sure that the package management tools are up-to-date first with “python -m pip install—upgrade pip setuptools wheel”.
Pip is very fast now and shouldn’t take more than a second if all dependencies are already installed. The only part that’s too slow is installing a package from source.
CFLAGS="-march=native -mtune=native" CONFIGURE_OPTS="--enable-optimizations --with-lto=full" pyenv install 3.11.1 --verboseAlso, I don't want my CI to fail randomly. https://github.com/python-poetry/poetry/pull/6297
Look at the number of issues in poetry. Some things are not backwards compat, some are broken. If you want reliable, KISS with pip tools.
That's my opinion.
I’ve tried PDM as well. It’s definitely nice. Part of me just wants to avoid “all encompassing” tools and use a specific tool for a specific job.
I used to have a dedicated folder to manage these environments myself, but seeing conda/mamba does a good job managing these; and they are also easier to load with command, so I switched to the managed approach.
Conda does not only work for Python, but many software necessary for our workflow that are written in C/C++/Fortran has been precompiled and uploaded to the conda-forge index as well.
./scripts/create-venv.sh # create venv using specific python version, update pip, wheel, setuptools, install requirements.txt
./scripts/freeze-requirements.sh # create tmp venv with requirements-base.txt and pip freeze > requirements.txt
./requirements-base.txt # dependencies with loose versions
./requirements.txt # frozen/locked dependenciesI just do it inside the container with pudb.
When you run a project in a container, you mount the projects directory into the container.
That is nothing my editor cares about. It does not even know it. It works on the files locally on the host os and that's it. The the files are executed somewhere inside a container is none of the editors business.
The system will have a bunch a bunch of (site) packages already installed, and hence you lose control.
I just run everything in a container and never need a virtual environment. Am I missing out on anything?
I don't experience any difference, whether I am inside a container or on the host os.
I did have some issues with some projects and want to revisit this soon.
Just do "docker run --rm -it debian:11" and you are ready to go.
Unfortunately as a developer it's more fiddly than that.
You need to mount your code into it. Then if you try commit changes to your code, you're a different user, so the commit is now messed up. Also you lost your dotfiles, so none of your aliases or other shortcuts work when you're in the container. You might need to connect to other services which now need to run in their own containers, and then they all need to talk to each other. Before you know it there's a whole litany of commands you need to run and configuration files you need to put together just to get started in the morning. Imo using containers on development machines is a big step backwards for developer productivity.
Virtual environment is just... cd in the directory. Run magic incantation to set that directory as the canonical one for your current shell. The end. IDEs will pick it up automatically.
The Python virtual environment and package management experience is still worse than pretty much any other programming language, but for me it's far and away better than using containers on a development machine.
I don't do that. I edit it on the host OS.
I just run the code inside the container.
I use containers regularly but adding them to a development workflow for a standalone python application I could just as easily run directly without having the additional complexity of navigating container boundaries always mystifies me.
docker run --rm -it -v $(pwd):/some/where debian:11
Most of the time I don't use plain Debian, but an image based on Debian which I have peppered with some convenience stuff.
I don't think it is practical for compute heavy scientific visualization. Setting up GUI with docker was a nightmare for me. There is also some overhead for CPU/GPU heavy code.
So you can install whatever you can apt-get/wget into it?
I can see that being a problem in restricted environments.