I just want to run this one Python script
die-welt.net
die-welt.net
It's then a (mostly) hermetic Python executable.
When I got it working, it felt like the first genuine 'hack' of my career. Something that's awful, novel*, and awesome at the same time.
* novel at least to me. It wasn't something I copied off stack overflow or a blog post.
Besides, it doesn't work on Windows, if you want a zipapp, shiv is a better alternative: http://shiv.readthedocs.io
This was hacking I was doing with a google/subpar (.par) file.
You might be running with your interpreter, but are you running with your standard library? The solutions out there will make sure to package up not only the interpreter and your files, but also any other files you import.
pip3 and virtualenv.
Both are pretty easy to learn. Everything else is pretty much an abstraction layer on top of one or both of the above.
That being said, I’ve been using Python for 16 years now and for me poetry is pretty close to nirvana. Not perfect, but really good. Handles virtualenv management, locking, dependency specification, etc etc.
Uhh, I’m struggling to think of anything else. It makes requirements.txt based projects feel barbaric and that will make you feel bad working with them?
1. Use something that manages virtualenvs + packages for you.
2. Don’t use something that manages virtualenvs + packages for you.
For 1 you have poetry or pipenv. For 2 you have… well virtualenv/pip. It’s really not that complex and I don’t get where this sentiment comes from.
You’ve already listed four separate applications: poetry, pipenv, virtualenv and pip. And if you look online you’ll also find Anaconda and probably several others. For a Python beginner, it’s confusing to try figure out the exact differences between them — let alone what exactly a ‘virtualenv’ is or why you would want to use it.
By comparison, Haskell — a language with a reputation for poor dependency management — has only two competing package managers, and these are practically identical. Most other languages have only one ‘official’ package manager. No wonder people find Python dependency management confusing!
So that’s the base level. You’ve got pip and virtualenv. Everything else builds ontop of that.
But hey it turns out it would be cool to iterate on stuff outside of Python itself. And maybe there are some useful things we can add to pip and virtualenv? And so Poetry and Pipenv are born.
It’s probably Stockholm syndrome but this seems pretty simple.
There are multiple PEPs about pip.
https://www.python.org/dev/peps/pep-0453/ makes it as official as it gets.
Source: I am a pip maintainer and a PyPA member.
Oh damn, well consider me schooled. Fair enough!
- pipx
- nix-shell (my current solution)
This feels a bit anti-intellectual to me. Like any other thing, it takes a bit of reading to figure our virtual environments, but its literally one line to create a new one one you know how (I almost always use Conda). I don't see how this is different than the setup used for any development, and I can't imagine it's a major impediment, certainly not a lasting one, to people working in python. You could always open a colab notebook if you really don't want to be bothered.
The problem is that depending on what article you read or what tutorial you follow you'll learn different things.
I recently wanted to write something using flask. I followed 4 different tutorials and all 4 did things differently (and none of them mentioned Conda or Poetry). I had a similar experience writing a Telegram bot.
Pyenv: tool to manage different python major/minor/patch versions.
Pipx: installs CLI/GUI tools in their own virtualenv and links them to ~/.local/bin. Way more convenient if you have commonly used tools
virtualenv: use to isolate python environments
Pip: used to install packages
Conda: basically does all the above, also handles linked libraries and a bunch of stuff. Annoying for deployment, but it's often the least worst option.
Python-N.M: use pyenv to manage your python installations. Way cleaner than brew. Never use system python for things.
Python2 thru 3.3: you really ought to be using py>3.6 by now, but it may not be your choice. I haven't in years but I think you can use pipx or "pip install --user virtualenv" to get the virtualenv binary. Then use virtualenv from there.
Py>3.3: use "python -m venv" to manage virtualenvs. Use pipx to install cli tools. Do everything in venvs.
On windows or if you rely on lots of academic packages: you are probably better off just using conda.
Also the pre-commit tool manages its own virtualenv cache, which is cool for making repos easy to bootstrap.
There's also poetry, but it's more new and if you are at the point in your python journey that you are asking these questions, then in my opinion it's likely to just muddy the waters more.
And if anyone is curious "why the hell is python this way? I don't have this problem with $LANGUAGE?" It's because when python fires up, basically a small miracle occurs that is able to locate and execute .py, .pyc, .zip, several other file types, directories, compiled libraries, and basically anything you want if you override the module machinery.
#!/bin/sh
""":"
# bash code here; finds a suitable python interpreter and execs this file.
for pyver in 3.6 3.7 3.8 3.9; do
which python$pyver > /dev/null 2>&1 && exec python$pyver "$0" "$@"
done
echo "No appropriate python interpreter found." >&2
exit 1
":"""The top reply had a great suggestion. Putting this at the top of a python 3.6+ required script results in a reasonable error for those with older /usr/bin/python interpreters, instead of a generic syntax error stack trace.
#!/usr/bin/python
0_0 # Python >= 3.6 is required import sys
if sys.version_info < (3, 6):
print('This script requires python 3.6+)
else:
from module import main
main()
If you want all of it to be in one file, you can use a zipapp: https://docs.python.org/fr/3/library/zipapp.htmlIf you don't want to make the zipapp manually, or want it to automatically extract everything on first run, use shiv:
Edit: Zipapps feel hacky to me. I get that not everyone would share that opinion.
All of this works fine with python 2.7 as long as you put the rest of your code in your main() entry point, which you should do anyway. So an f-string would cause no problem.
Also, "informative stack trace" is only informative to another dev, and an experienced one at that, given how many juniors I encounter that can't read a traceback.
Indeed, you script users may not be devs, and for those, a traceback would be confusing. For this reason, a script should only output a traceback in dev mode, but in production, if the dev flag is not set (-X dev), you should never see a traceback. It's a good practice to suppress it with sys.excepthook, and log it into a file instead.
Using a syntax error like you do for this purpose IS the big hack.
Now I'm not against a hack. I can see the value of it for your own scripts, of things that you code quick and dirty style. After all, I don't always want to have a clean entry point and more than one file.
You keep repeating the same thing a if you ignored this essential part.
Well, yes, and you keep insisting that everything is fine, as-is. There was a missed opportunity here.
import sys
if sys.version_info.major < 3:
raise Exception('Need Python 3 or higher')
EDIT: correction, sys.version_info.major is only there since Python 2.7. For older versions sys.version_info[0] should be used.Right, but if there's anything below that that would cause a syntax error, like an f-string, that Exception you created never happens. The end user just sees a syntax error.
It's unfortunate there's not a better way to handle it, especially with so many people still having /usr/bin/python linked to a 2.x version.
f"Python 3.6+ is required" #!/usr/bin/env python
here?Or theoretically also the other way around.
#!/usr/bin/env python
If you do, then `python2` or `python3` will be around, and you should specify them explicitly instead of `python`. I'm happy to be corrected on this one, but I always slap this at the top of my scripts: #!/usr/bin/env python3
I'm not aware of anywhere that wouldn't invoke a Python 3 interpreter if one is installed.The original post just seems like one big faff.
https://www.python.org/dev/peps/pep-0394/
"The python command should always invoke Python 2 (to prevent hard-to-diagnose errors when Python 2 code is run on Python 3)."
So there was indeed a time where the mentioned scenario was possible.
Edit: love those people who got to downvote reality. At no point was python pointing to either 2 or 3 will-nilly, but there was always a PEP specifying best practice behaviour.
The current version of PEP 394 says:
] If the python command is installed, it is expected to invoke either the same version of Python as the python3 command or as the python2 command.
] Distributors may choose to set the behavior of the python command as follows:
python2,
python3,
not provide python command,
allow python to be configurable by an end user or a system administrator.
The part you quoted is from the section "History of this PEP". Just before your quoted text is:] This PEP originally provided a very simple mechanism ...
and just after it is:
] However, these recommendations implicitly assumed that Python 2 would always be available.
This historic advice is no longer recommended.
The change was made on July 5, 2019 - https://github.com/python/peps/commit/ae932bd6fd2c493d7d64ce... .
Command 'python' not found, did you mean:
command 'python3' from deb python3
command 'python' from deb python-is-python3
https://packages.ubuntu.com/focal/python-is-python3Of course, if your script will run under both Python 2 and 3, I don't know why you would want to use these contrivances to simply give preference to Python 3.
Performance gains, preferring future-looking development, and better error messages.
The article is literally about this scenario, which is why I'm also confused. Yes, `/usr/bin/env python` is a bad idea IF your script depends on a certain version, but that's exactly what the author doesn't need!
Since Debian 11 and Ubuntu 20.04 LTS it's a matter of installing the matching package:
python-is-python2 [1] for having /usr/bin/python link to Python 2.7, python-is-python3 [2] for having /usr/bin/python link to Python 3.x or none of them to avoid having /usr/bin/python at all. The latter one will still result in a working system, as all Python scripts shipped by Debian/Ubuntu nowadays explicitly specify /usr/bin/python2 or /usr/bin/python3.
This is despite guidance from PEP 0394[1]
When designing things for 'just work' scripts you have to take esoteric environments into account.
There is no guarantee that '#!/usr/bin/env python' will not error in an environment with only python3 installed.
So I do wonder how many distros (and manual install workflows) for python could cause this to happen if python2 got uninstalled from a system that had both, where python get removed, rather then updated to point to python3.
(Ubuntu LTSs and macOs are an example ...)
eval 'exec /usr/bin/perl -w -S $0 ${1+"$@"}'
if 0; # not running under some shell
See https://stackoverflow.com/questions/9143726/what-means-not-r...Probably too late for that; if you add this, you'are just complicating things: scripts then have to worry about pythonx not existing.
The env program could have a feature to help here. GNU Coreutils could adopt an env feature whereby it searches for multiple programs. Because of argument limits in the hash bang command line this would have to be encoded as one argument.
#!/usr/bin/env python|python3
So here, env sees there is a pipe in the middle of the program name. It splits the program name into two, and first searches the PATH for python. If that is not found, it searches the path for python3. Alternately, rather than giving preference to the left item, it could prefer the first one that occurs in the PATH. It could search the PATH once, stopping whenever any of the indicated names is found.Or, how about supporting regex in env, if the first character is circumflex:
#!/usr/bin/env ^python3?
The advantage is that the specification is condensed. On some systems, there is a strict limit, like 32 characters, into which the hash bang line has to fit, so this could be helpful against that. The problem with that is that *env now has to read directory entries in user space to do the matching; it can't do a simple access() test on the name in every directory. This wholesale directory reading fails to take advantage of whatever name lookup caching mechanisms there are in the underlying kernel.The issue is that on some systems /usr/bin/python does not mean "any version of python", but specifically Python 2.
And so, in those systems, if the Python 2 package is not installed, then there is no /usr/bin/python.
They can't easily change the meaning of /usr/bin/python, because the distro users adopted to the convention and depend on unqualified python being python2, and python not being found as an indication that Python2 is not installed.
Maybe /usr/bin/sensible-python or /etc/alternatives/python?
The Debian-style alternatives mechanism could work here. I had that in the back of my mind when I wrote the comment, but I wasn't sure whether or not it's already buggered in some way by the Python2/3 thing.
(Like if there are multiple Python2 packages somehow, using /etc/alternatives/python to have the semantics of which of the two is the preferred Python2.)
If the /etc/alternatives/python name is available, it could be used this way: the preferred python (for running code that isn't specific to any python version).
It works this way for some existing programs. For instance /etc/alternatives/yacc could be GNU Bison or Berkeley Yacc. Either implementation does not handle some inputs that the other does, and there is a POSIX yacc subset of the yacc language that should work with both. If you have a very plain yacc file that conforms to POSIX and stays away from corner cases in the spec, it should just work with the yacc command.
If those two things don't cause the script to be executed by Python then frankly the user's system is broken and I'm not going to worry about it. (Depending on what you're writing, `#!/usr/bin/env python` may be a better option, so that you respect the PATH - including venv's.)
* Well, they were compatible, but I no longer test against Py2.
Running FreeBSD, one of my pet peeves is folks who assume /bin/bash is going to exist everywhere.
But all my knowledge about that comes from a small drama some times ago when a debian maintainer wanted to remove "which" from its current package (see https://lwn.net/SubscriberLink/874049/bf89a969ed3dde87/).
[1] https://searchfox.org/mozilla-central/source/mach
Pretty neat, although its support for Windows hosts seems to rely on Powershell (or dispatches completely to anything other than Python. Not sure about that)
There are still shops with computers running windows XP, performing various scripted automation or batch tasks.
The idea that some running system will be taken offline just because the language is old is incompatible with a lot of real world needs.
Having said that, 2.7 still has commercial third-party support, e.g. from ActiveState. Which, presumably, will be around for as long as there are enough users willing to pay for it.
CentOS 7, for instance, still has 3 more years of patch support
cat <<-DOC
line 1
line 2
line 3
DOCExample:
foo(){
: "This is a docstring for the foo() function"
bar --verbose | baz --quiet
}
For those who don’t know: ":" is a shell built-in command which does nothing and ignores all arguments. It was the way in which shell scripts had comments before the modern syntax of using # for comments was introduced.Colons, “:”, in batch files were, IIRC, for labels, i.e. targets for “goto” statements.