And stop putting everything on the planet into a god damn virtual environment.
Right, but then just use pip install --user instead of a virtualenv. Actually --user is the default now when running pip install from non-root user, so, just pip install as a user will work and not mess with the system packages, just don't do sudo pip install.
Not to mention that installing every dependency in the same environment is a recipe for both disaster with version conflicts and bloat when you don't really know which packages belong to which applications.
However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, it all depends how well the said packages are maintained in which case just upgrading them should fix whatever problem.
Distro packages are done in a way where it's hard to get conflicts. I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often.
Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't. You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word.
Exactly, because the packages were created/updated at a point in time were they were compatible: a distro ships python 3.x with packages of python modules that are compatible with python 3.x.
Now if you as a user want to use a python package that works with python 3.y, you either have to wait, or install python 3.y, or use a container of python 3.y.
Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work.
> Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't.
Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;)
> You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word.
Let's agree to disagree right there:
- in your point of view: this is a problem in pip
- in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning
That is impossible, it will never happen. Packages will have different maintenance cycles, some will get deprecated, others abandoned... You can't base your upgrade policy on an impossible situation.
> Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;)
I know, but pip doesn't track all of those all the time. An example that has happened to me multiple times: install package X that depends on Y <= 1.0. Good, pip installs the proper version. Now, on another command, install package Z that depends on Y >= 2.0. Pip will install Y >= 2.0 and won't care that package X is now broken.
> - in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning
Regardless on your views on pinning, it's a problem in pip. A version conflict should be reported as an installation failure, not allow you to continue.
And again, version restrictions will always be there. The "live at master" philosophy only works for small groups of similar output capacity. It won't work for an ecosystem as wide as Python. Even without version pinning, you'll still have packages breaking because another one was updated. Sometimes it will be necessary, such as for example a package dropping support for a feature that another one needs.
You realize that if that was "impossible" and "never happening", then absolutely no Python environment would be working ever?
> A version conflict should be reported as an installation failure, not allow you to continue.
Please don't make this mandatory.
> Pip will install Y >= 2.0 and won't care that package X is now broken.
Fine, I'll just quickly fix X and open a pull request, like I probably did a hundred times, then eventually if necessary deploy my fork with the fix meanwhile they do their maintenance release.
Should we not take the responsibility of the dependencies we use and contribute back??
> The "live at master" philosophy only works for small groups of similar output capacity. It won't work for an ecosystem as wide as Python.
I don't understand why, but I'm talking about "live at latest release", not "at master".
> Sometimes it will be necessary, such as for example a package dropping support for a feature that another one needs.
Then the package can just paste the code of the feature in their own, until a new lib does it, I remember having to do that twice in 20 years (except I didn't just "paste" it, but implemented a much smaller version).
Overall, it seems my approach produces versions of my software and software that I depend on that are compatible with all versions because you can always use an earlier version if you really want to deploy on an old python or whatnot, whereas you approach leads to broken packages, tech debt, and blaming the package manager.
No, it means that you can't be fully sure that upgrading everything to the latest version is not going to break anything. And right now, you can't. You can only do that in controlled environments (say, a distro's official package repositories), not with Python packages.
> Please don't make this mandatory.
It is mandatory in quite a lot of package managers. APT, for example, will refuse to install packages with conflicting versions or breaking things. It's better to fail early with a clear message than to have a later failure where the cause is unclear.
> Fine, I'll just quickly fix X and open a pull request, like I probably did a hundred times, then eventually if necessary deploy my fork with the fix meanwhile they do their maintenance release.
That's optimistic. What about detecting which package is causing the issue? What if the fix is not quick? What if the package developers are working on compatibility but it's going to take time?
> I don't understand why, but I'm talking about "live at latest release", not "at master".
Problem is similar. You can't live at latest release with software coming from wildly different developers, with wildly different policies on compatibility, versioning, breaking changes, bugs...
> Then the package can just paste the code of the feature in their own, until a new lib does it, I remember having to do that twice in 20 years (except I didn't just "paste" it, but implemented a much smaller version).
Again, pretty optimistic. You won't be able to do that with all packages. For example, right now I have a project that needs to work on Python 3.6 (among other packages). Latest numpy versions dropped support for Python 3.6, so the project needs to restrict numpy versions to maintain compatibility. I can't just take the latest numpy and patch it for Python 3.6.
> Overall, it seems my approach produces versions of my software and software that I depend on that are compatible with all versions because you can always use an earlier version if you really want to deploy on an old python or whatnot, whereas you approach leads to broken packages, tech debt, and blaming the package manager.
You also spend time doing maintenance and debugging on new installations to debug and fix compatibility issues, when those upgrades might not bring any value to the product. In my case, I do that debugging and fixing in a controlled environment, only when I decide to upgrade the packages, and maybe revert/pin the ones that don't have quick fixes. That's the difference. Once a given package is released/packaged for distribution, I want dependencies to be fixed so that every new installation works, and doesn't fail if some developer decided to break things that day.
Well, it's still what you do every time you create a new project.
> It is mandatory in quite a lot of package managers.
Ok but many packages won't be installable at all anymore, because people pin different versions instead of maintaining their packages by continuously integrating upstream releases.
> You can't live at latest release with software coming from wildly different developers, with wildly different policies on compatibility, versioning, breaking changes, bugs...
Well I didn't know can't so I did.
> I can't just take the latest numpy and patch it for Python 3.6.
Instead, you should be upgrading your code to support the newer numpy version. Seriously, try it out, you'll be spending less effort at the end of the year.
> You also spend time doing maintenance and debugging on new installations to debug and fix compatibility issues, when those upgrades might not bring any value to the product.
I spend less time and more spread over the year, bringing value to the whole ecosystem of packages which are bringing value to my product, and bringing value to myself as I develop new products with the dependencies I like and support as such.
> In my case, I do that debugging and fixing in a controlled environment
I wait for CI or users to report that a new release isn't compatible so I can fix it as soon as possible, in which case we temporarily pin versions, instead of piling up tech debt until it's so huge the whole project becomes trash.
That's what I call taking responsibility with the dependencies that you include in your product.
Yep, and it is a pain in the ass when the package you want to use creates a dependency conflict. Luckily, at that stage I can still search for an alternative with little to no cost.
> Ok but many packages won't be installable at all anymore, because people pin different versions instead of maintaining their packages by continuously integrating upstream releases.
And some packages won't be installable anymore because they use deprecated APIs, or were only compatible with Python 2, or expected certain files/folders/programs to be present on the system and they aren't anymore. Packages go into abandonware all the time, removing version pinning doesn't fix that.
> Instead, you should be upgrading your code to support the newer numpy version. Seriously, try it out, you'll be spending less effort at the end of the year.
Thank you for telling me this, I will be relaying to my clients that they need to upgrade their systems to newer Python versions to get new features, surely that will go well.
> I wait for CI or users to report that a new release isn't compatible so I can fix it as soon as possible, in which case we temporarily pin versions, instead of piling up tech debt until it's so huge the whole project becomes trash.
Well, I prefer to not have my program crashing in client environments because someone decided to update a package and break things.
> piling up tech debt until it's so huge the whole project becomes trash.
Waiting to release on new dependency versions until those versions are tested is not tech debt. I mean, semantic versioning was done with this purpose precisely: to signal which upgrades are just bug fixes and you should upgrade ASAP, which ones are new features so you can upgrade without too many problems, and which upgrades break compatibility and you should test it thoroughly in case something broke.
If living at latest release works for you and dealing with dependency issues on new installations is not a problem, then go. But other systems will require that a certain release installs dependencies on a consistent manner. Some of my dependencies are automatically upgraded and tested before release, others are pinned because they always break things and I only upgrade manually. But once a version of a package is released, that one goes with version pinning because I prefer to avoid the risk of a client installing the software and having to explain them than "oh, it's just that this dependency released a new version that broke things and we didn't specify the version that our software needed".
Abandoned dependencies should be treated like dropped features: re-implement them in your own code or create another library.
> Thank you for telling me this, I will be relaying to my clients that they need to upgrade their systems to newer Python versions to get new features, surely that will go well.
Very funny, but seriously they should have an upgrade path, or install their machine once and then never do a single upgrade, but again I'd say this is just tech debt piling.
> Waiting to release on new dependency versions until those versions are tested is not tech debt.
Not having your own tests is tech debt thought. I wouldn't rely on tests made by others.
> I mean, semantic versioning was done with this purpose precisely: to signal which upgrades are just bug fixes and you should upgrade ASAP
Of course I agree, the problem is that maintainers tend to not upgrade as often as they should, like until they really have too (another dependency upgrades a shared dependency). Again I see this as a maintainance problem rather than a package manager problem.
> If living at latest release works for you and dealing with dependency issues on new installations is not a problem, then go.
I'm not saying I never had a problem of course I do, I'm saying that having these problems has a better cost/benefit ratio at the end of the year, because: 0. they are smaller, dealing with one BC break at the time, and 1. they are spread over time because I integrate upstream releases continuously instead of waiting for the day I want to upgrade everything.
> "oh, it's just that this dependency released a new version that broke things and we didn't specify the version that our software needed"
I'd rather say "oh, it's just that this dependency released a new version that we are implementing support for as we speak, here's the command you can run to fix it meanwhile: ...".
But if you don't want to have that problem, just run your CI periodically, make sure the first thing you do in the morning is checking that nightly did actually pass tests, as you're talking about paid software maintenance, I'm talking about both paid and volunteer so that's why I'm including "wait for a user to report" but that doesn't apply for paid maintenance of course.
For me we're seeing the discussion between two fundamentally opposite approaches, one being defensive and the other being offensive / actual continuous integration. From my experience, the offensive strategy offers a better cost/benefit ratio at the end of the year.
This seems almost as bad as system python. I suppose it's fine if you only work on one thing, but as soon as you don't, your dev environment will become chaotic and lots of confusing WFM will happen. E.g. this is why npm has a separate node_modules folder for each project.
This sacrifies time and disk space and hides tech debt under the carpet, I'd rather have a solution like `pip upgrade` that would upgrade all packages and fix the environment, like `pacman -Syu`, but people would have to stop pinning versions and actually maintain their codebases and the dependencies they uses.
Virtual environments is the mechanism by which you do that in a non-silly way. How do you think people cope with a dependency that has a bug introduced in the most recent version?
Go ahead and install pip3 from Python 3.6. use that to install pip3 for 3.8, and try using apt. Backup first.
Oh dear... What kind of monster does this?
You'll get pip3 from Python 3.6.
apt install python3.8
You'll get system-level python3.
Use pip3 in 3.6 to install the latest pip3 that requires Python 3.8, and drops that package in the dist-packages folder, which is shared between system Python runtime versions.
Behold as further attempts to do something sane with APT blow up, because now you have 3.8 packages on 3.6's runtime path.
I've lost weeks to this particular brand of distro daftness. I thought that surely no one would do that .. Alas...
If you don't want someone to mess with it, name it as something other people will not regularly have a reason to muck with. One concerted effort to ship distro's with a "sys-python" symlink instead of /usr/bin/python, and for all things currently using the newly minted sys-python stuff can be free of wayward travelers and system sculptors like me mucking with things.
No, pyenv is not the answer. Naming things is.
And trust me. I get it. It's a rite of passage learned in due time, yada-yada. Got it. If we ever want to get this stuff easier to use though, we have to be willing to help draw clean distinctions between this and that. Virtual environments don't do that until it is far too late and a beginner is already in and over their head in breakage.
In my experience, Pyenv is almost never the answer.
>The package maintainers ship 3.6 by default for a reason and they know better than you! /s
Maybe then! Not anymore! It really does infuriate me because it breaks every aspect of the principle of least surprise and conventional software packaging practice.
I mean... I understand how hypothetically this could in theory make some sense because a distro might naively expect to ship the same package version for different Python versions and so all files could be the same except for native extensions, which are disambiguated by implementation name, so theoretically one could imagine a position where this could be conceived of as a thing one could do to save some space in case someone installs the same package for multiple Python versions.
But in reality? Jesus fucking christ. However, I don't see how this can be blamed on Python. This isn't something that "just happens because Python" and is certainly far, far away from any kind of default Python configuration, where all default paths are below the prefix, which contains the Python version. This is something that can only happen because someone went extraordinarily far out of their way to shoot someones else's foot off.
It's definitely the distro's fault (cough Ubuntu cough).
>This is something that can only happen because someone went extraordinarily far out of their way to shoot someones else's foot off.
Ding ding ding.
As far as I can tell, they're a strict requirement, if you ever work on more than one project and want to isolate the dependencies (because you need different versions of the same package, because you want to ensure you've tracked all deps, whatever).
Is it that you don't isolate dependencies between your python projects at all, or is there some other solution you prefer?
In summary: bin/run-app -> bin/venv-python -> .venv/bin/python -- I'm not sure how to make it simpler than this. The cost is perhaps in disk space, but I don't care about that.
Or don't do the above and deploy via docker and use pip to install to the system python in the image.