Hacker News Reader

Comment by thu | Hacker News Reader

The biggest gripe I have with using Python for rich command-line tool is the startup time. One of the first reason I like to write a nice command-line tool when I start a project, say foo, is to be able to do `foo --help` to quickly see and remember what the project can do (I have a very bad memory and doing this makes it possible for me to jump back faster to a project, even well documented. I can forget what I was doing in just a few days and so I add a lot of small commands).

In short running `foo --help` should be instant and if it loads all its modules to list the different sub-commands and their respective description it is really too slow.

A possibility is to cache some information (e.g. generate a text file or a small Python script).

View on HN

Hm, it'd be pointless to argue this point (if you think it's slow then it's slow) but I've created many a Python CLI and have never noticed them being slow to start. Perhaps it's a case of importing certain modules globally instead of on a per-function basis?

See my answer to coldtea. Yes you can organize you're code to help making it load faster but I don't think it is fast enough. Maybe my terminal is slow, maybe my machine is slow (and maybe I'm overly sensitive to the problem) but the difference is telling.

>The biggest gripe I have with using Python for rich command-line tool is the startup time.

What startup time?

I even have a python script running on every shell prompt drawing (that checks mercurial on top of starting the python interpreter), and the latency is negligible.

I just tried on an app at work (Django app):

    > time python manage.py --help
    real	0m0.473s
    user	0m0.240s
    sys	0m0.148s

Half of a second is very noticeable (and annoying). I had similar perception in my previous work.

You can try to only load enough to display the help text without really loading everything, but that doesn't work that well (you have to organize things differently, and `--help` requires to load a lot of stuff. `subcommand --help` needs less but it still has to see if the subcommand exists).

As a reference:

    > time python -c 'print "hello"'
    hello

    real	0m0.041s
    user	0m0.024s
    sys	0m0.012s

    > time echo hello
    hello

    real	0m0.000s
    user	0m0.000s
    sys	0m0.000s

Actually even the first (0.041s) is not instant, while the second one does (I mean as I perceive it visually).

This is why I use Lua for command-line tools. Here's one such program that makes no attempt to organize code to make --help load faster:

  real  	0m0.006s
  user  	0m0.000s
  sys   	0m0.004s

Still, your second example is an order of magnitude faster than the example with Django. That's likely because Django loads the world on startup, parses all your model files, checks your database settings, etc.

To be fair, Python without loading site packages is pretty damn fast. Definitely faster than most other things loading up.

  $ time python -S -c 'print "Hello World!"'
  ...
  real	0m0.007s
  user	0m0.004s
  sys	0m0.003s

That's about the same speed it takes my system's cp command to show me an error page:

  $ time cp -fail
  ...
  real	0m0.005s
  user	0m0.003s
  sys	0m0.002s

That is your Django app, not Python itself, putting in that overhead.

Calling a shell built-in isn't really fair -- you should at least compare to /bin/echo (on my system the relevant times are [edit: 0.001s] for echo, 0.007+0.004=0.011 for python -S -c "print 'hello'". Without dropping site-packages (without the -S) python jumps to 0.014+0.012=0.026 -- marginally above 200 ms which I suppose is a perceptible difference between instant, and not-quite-instant).

Sadly, both pypy and python3 are slower than python2.7. Also worth nothing that the sys time fluctates for me -- in other words when it is > 0.00s it doesn't appear to have anything to do with the command run.

> Half of a second is very noticeable (and annoying). I had similar perception in my previous work.

That's because it has to load django and the whole django project. Here's click:

    > time python test.py --help
    Usage: test.py [OPTIONS]

    Options:
      --help  Show this message and exit.
    python test.py --help  0.06s user 0.02s system 95% cpu 0.078 total

and argparse:

    > time python test.py --help
    usage: test.py [-h]

    optional arguments:
      -h, --help  show this help message and exit
    python test.py --help  0.03s user 0.02s system 93% cpu 0.054 total

reference:

    > time python -c 'print "hello"'
     hello
    python -c 'print "hello"'  0.01s user 0.01s system 88% cpu 0.025 total

I'm completely clueless about this, but would it be possible to have a persistent python interpreter always running in the background somehow, and submit the commands to it?

If loading the rest of the code and initialization time really matters, yes. But then you're back to square one: you now have to create a CLI to access the server and that new CLI must load quickly. Which would also be the case if you just wrote it that way first, then loading the rest of the code once the command is determined.

And if you're in some project similar to Django, you would have first to query the server to know the possible subcommands...

So what you say makes sense when other parts of the code are slow to initialize and must be accessed frequently, not for the CLI itself.

You can keep ipython kernel running in background if you want. Quick unscientific test showed that it didn't reduce ipython console startup time significantly. But it should be possible to make a leaner console to connect to the kernel.

edit: my test was flawed, with more accurate test there is almost a full second time difference:

    $ echo -n | time ipython console --existing kernel-29793.json
    1.06user 0.06system 0:01.52elapsed 74%CPU (0avgtext+0avgdata 29968maxresident)k
    0inputs+64outputs (0major+19245minor)pagefaults 0swaps
    $ echo -n | time ipython console
    1.07user 0.08system 0:02.40elapsed 48%CPU (0avgtext+0avgdata 30264maxresident)k
    0inputs+72outputs (0major+20538minor)pagefaults 0swaps

of course two seconds is ridiculously slow either way

I think you want a pool of python interpreters read to go.

Reply on news.ycombinator.com